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EXECUTIVE  SUMMARY 

One  purpose  of  the  study  was  to  document  the 
differences  in  the  development  of  hardware  and  software  for 
USAF  weapons  systems.  A  further  purpose  was  to  document  the 
functions  presently  being  performed  by  the  buying  activity 
(SPO)  and  by  contract  administration  services  (CAS) 
personnel  in  software  acquisition.  Another  purpose  of  the 
study  was  to  document  the  technical  functions  which  should 
be  performed  by  SPO  and  CAS  for  more  effective  management  of 
software  acquisition.  A  final  purpose  of  the  study  was  to 
document  personnel  skills  (for  CAS)  and  policy  changes 
needed  to  support  the  recommended  technical  functions. 

(For  convenient  reference,  the  headings  of  this  summary 
are  identical  to  the  chapter  headings  of  the  full  text.) 

General  Differences  in  Hardware  and  Software 

An  increasing  amount  of  the  dollar  and  manpower 
expenditure  for  weapons  system  development  is  going  into 
software.  The  following  figure  shows  the  dramatic  shift  for 
development  personnel  over  two  decades. 
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FIG.  ONE 


GROWTH  IN  SOFTWARE 
PERSONNEL 
(Glas«man,1982) 


Currently,  control  aspects  of  weapons  systems  are 
coming  to  be  dominated  by  computer  software  and  the 
appropriate  methods  and  approaches  for  the  acquisition  of 
weapons  system  software  bear  subtle  and  critical  differences 
from  parallel  methods  for  hardware. 

Although  the  engineering  principles  applicable  to 
software  and  hardware  are  similar,  the  tremendous 
differences  in  the  essential  nature  of  the  two  products 
requires  that  the  methods  for  applying  these  principles  be 
different.  Since  these  differences  and  organizational 
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tendencies  to  deal  with  them  less  effectively  than  desired 
form  the  basis  for  this  study,  the  differences  are  discussed 
in  the  text  in  some  detail.  Table  One  shows  the  nine  areas 
of  software  versus  hardware  similarities  and  differences 
which  are  discussed. 


MNE.AREAS  OF  DIFFERENCE 
IN  SOFTWARE  AND  HARDWARE 

(Overview) 


o  Physical  Existence 

o  Physical  Dimensions 

o  Measurability 
o  Standard  Elements 
o  Life-Cycle  Relationships 
o  Design  Precision 


o  Cost  of  Design  Correction 
o  Quality  Assurance 
o  Comprehensive  Testing 


TABLE  ONE 


The  text  includes  suggested  applications  in  each  of 
these  nine  areas  for  the  management  of  software  acquisition. 
However,  two  recommendations  based  on  these  differences  are 
especially  clear  and  potent: 

1.  Early  Involvement  and  Early  Error  Correction 

In  spite  of  any  theoretical  positions  or  current 
management  philosophies,  the  orientation  of  DOD  to 
quality  assurance  is  understandably  hardware-dominated, 
and  as  such,  places  its  greatest  manpower  effort  in  the 
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quality  evaluation  of  end  products.  In  the  case  of 
hardware,  this  is  probably  a  good  strategy.  In  pre- 
production  and  production  development,  the  government 
definitely  participates,  but  puts  the  large  burden  of 
technological  manpower  on  the  contractor.  But,  in  the 
final  phase  of  quality  assurance,  the  government 
greatly  steps  up  its  participation  to  assure  the  final 
end-product,  before  it  is  accepted. 

This  orientation  is  inappropriate  for  software. 

The  most  critical  activities  in  software  take  place  in 
the  design  and  testing  phases.  Not  only  is  this  in 
contrast  to  hardware  orientation,  but  the  production 
phase  for  software  may  even  be  considered  virtually 
nonexistent .  This  is  not  to  say  that  the  production  of 
copies  of  software  is  trivial.  Reproduction  of  tapes, 
disks,  etc.,  can  be  expensive  and  time  consuming  and 
must  be  quality  assured  like  other  production 
processes . 

In  place  of  a  significant  production  phase  for 
software,  there  is  instead  the  software  maintenance 
phase.  This  is  far  from  being  a  phase  where 
corrections  to  software  might  be  made  on  a  cost- 
effective  basis,  thus  taking  a  marginally  functional 
system  and  "tweaking  it  up"  at  reasonable  cost. 

Instead,  the  maintenance  phase  for  software  has  been 
repeatedly  shown  to  be  (Fairley,  1985) : 


RELATIVE  COST  OF 
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(a)  the  most  costly  phase  of  the  software  life 
cycle,  and, 

(b)  the  life-cycle  phase  in  which  any  given  error 
costs  the  most  to  correct. 

For  an  illustration  of  the  cost  of  correcting 
errors  at  different  points  in  the  Life-Cycle,  note  the 
following  figure. 


ECONOMY  OF  EARLY  ERROR 
CORRECTION 
(Martin, 1985) 


# 
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2 .  Early  and  Intense  Attention  to  Testing. 

In  hardware  systems,  testing  can  usually  be  nearly 
exhaustive,  if  desired.  Certainly,  there  are  trade¬ 
offs  and  not  all  testing  may  be  desireable  relative  to 
some  other  cost  factors,  but  testing  of  components  and'  y  \ 
complete  systems  is  generally  within  option.  The  case 
of  software  is  quite  different.  In  the  case  of 
computer  programs  that  are  large  enough  to  be 
significant  (and  the  programs  involved  with  weapons 
systems  are  very  large) ,  their  complexity  is  so  great 
that  even  automated  testing  by  intelligent  computer 
programs  cannot  permit  exhaustive  testing.  The 
following  diagram  illustrates  dramatically  how  the 
complexity  of  a  conceptually  fairly  simple  module 
quickly  leads  to  impossible  requirements  for  exhaustive 
testing. 


FIG.  THREE 

OVER  206  TRILLION  UNIQUE  PATHS 
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The  capability  for  the  program  to  be  effective  in 
the  test  phases  must  be  laid  down  and  assured  from 
the  earliest  design  phases.  Thus,  not  only  is  it 
cost  effective  to  correct  problems  in  the  design 
phase,  when  feasible,  but  it  is  also  necessary  to 
assure  the  quality  of  testability  from  the 
earliest  design  phases.  Otherwise,  the  test  phase 
can  be  explosive  in  terms  of  cost,  schedule,  and 
morale  (Evans,  1984) . 

Life-Cycle  Differences 

The  main  differences  between  software  and  hardware  in 
contract  administration  can  best  be  viewed  in  terms  of  the 
weapons  system  life  cycle  (WSLC)  and  the  software 
development  cycle  (SDC) .  The  overall  WSLC  consists  of  four 
sequential  and  essentially  discrete  phases: 

*  Concept  Exploration 

*  Demonstration  and  Validation 

*  Full-Scale  Development 

*  Production  and  Deployment 

Within  the  phases  of  the  WSLC,  hardware  and  software 
have  development  life  cycles  which  are  somewhat  comparable. 

(NOTE:  THE  PHASES  ARE  SIMILAR  IN  FUNCTION  AND  HAVE  SIMILAR 
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NAMES,  BITT  THEY  OPERATE  VERY 
See  table,  following.) 


SOFTWARE  DEVELOPMENT. 

CYCLE 

1.  System/Software 
Requirements  Analysis 

2.  Software  Requirements 
Analysis 

3.  Preliminary  Design 

4.  Detailed  Design 

5.  Coding.  Unit  Test  and 
Computer  Software 
Component  Integration 
Testing 

6.  Computer  Software 
Configuration  Item  Testing 

TABLE  ■ 


DIFFERENTLY  WITHIN  THE  WSLC. 


UMQmREJ2EYElQRMENT 

CYCLE 

1.  System/Hardware 
Requirements  Analysis 

2.  Hardware  Requirements 
Analysis 

3.  Preliminary  Design 

4.  Detailed  Design 

5.  Fabrication 

6.  Hardware  Configuration 
Item  Testing 


The  terms  used  here  are  self-explanatory  and  the 
sequences  are  intuitively  meaningful,  as  well  as  the 
parallels  in  software  and  hardware  (some  kind  of  parallelism 
should  be  present,  since  they  develop  as  linked  subsystems 
of  a  parent  system) .  But  here  is  one  of  the  most  critical 
differences  for  contract  administration:  phases  of  the 
hardware  development  cycle  are  expected  to  proceed  in  a 
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linear  fashion  with  virtually  no  overlap  —  by  contrast,  the 
phases  of  the  software  development  cycle  are  expected  a 
priori  to: 


*  Overlap  one  another  significantly 

*  Proceed  nonsequential ly  with  an 
indeterminate  number  of  loops  between 
successive  phases  (this  implies  possible 
paths  from  phase  6.0  clear  back  to  phase 
1.0)  . 

*  Overspan  the  phases  of  the  WSLC  (the 
hardware  cycle  is  expected  to  be 
finished  complete  within  a  given  phase 
of  the  WSLC) . 

(It  should  be  noted  that  both  the  hardware  development 
cycle  and  the  software  development  cycle  are  likely  to 
appear  in  somewhat  modified  forms,  as  is  appropriate,  in  the 
first  two  phases.) 

The  following  figure  attempts  to  illustrate  the 
complexity  of  these  relationships. 


hardware  cycle 


FIG.  FOUR 


COMPLEXITY  OF  SOFTWARE 
DEVELOPMENT  CYCLE 


Dennis  Blosser  &  Associates 
Final  Report 
Page  x 


Training  and  Selection  of  Software  Specialists 

Personnel  designated  as  software  focal  point  usually 
have  some,  but  typically  minimal  or  somewhat  irrelevant, 
training  and  background  in  software.  Although  the  software 
focal  point  is  not  necessarily  supposed  to  be  highly 
competent  in  software,  this  person  is  typically  looked  to  in 
their  organization  for  leadership  and  technical  competence 
in  software  problems.  NOTE:  There  appears  to  be 
considerable  difference  of  opinion  on  this  from  AFSC, 
HQ/AFCMD,  and  AFPRO  viewpoints.  If  they  are  indeed  to 
fulfill  these  expectations,  they  should  be  provided  with: 

*  Development  of  their  leadership 
abilities  through  training 

*  Development  of  their  leadership  roles 

*  Technical  training  (for  them) 

*  Technical  training  (their  support  for 

their  colleagues) 

It  is  clear  that  training  and  personnel  development 
needs  for  software  duties  are  strong  throughout  the  contract 
administration  organization.  Hardly  anyone  is  expressing 
competence  to  do  this  job.  The  few  who  have  strong 
competence  know  there  is  much  more  than  they  can  do. 

Nondeliverable  Software 

When  nondeliverable  software  is  used  in  the  development 
and  production  of  weapon  systems,  we  have  to  ask  the 
question: 
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*  What  are  the  consequences  if 

nondeliverable  software  does  not 
function  properly? 

Obviously,  the  consequences  could  range  from  trivial  to 
very  serious  both  in  terms  of  cost  and  safety. 

This  leads  to  the  conclusion  that,  even  though  seme 
support  software  is  nondeliverable,  it  still  requires 
careful  management  and  surveillance  from  both  contractors 
and  the  government.  Currently,  there  is  little  attention 
being  paid  to  the  various  aspects  of  management  of 
nondeliverable  software.  Notably,  one  approach  to  the 
management  of  nondeliverable  software  did  surface  with  some 
consistency.  This  is  a  report  by  Major  George  Trevor  which 
surveys  the  concerns  of  nondeliverable  software  and  offers 
some  guidelines.  A  draft  version  of  this  report  is  included 
in  the  Appendix. 

It  is  hypothesized  that  the  lack  of  attention  to 
nondeliverable  software  comes  from  four  sources: 

(1)  The  very  heavy  and  specific  focus  on 
deliverable  software, 

(2)  Existence  of  clear-cut  standards  and 
procedures  for  deliverable  software  that  are 
lacking  for  nondeliverables, 

(3)  Assumption  that  if  it  is  nondeliverable,  its 
development  cannot  be  important  enough  to 
warrant  management  effort, 

(4)  Assumption  that  if  the  nondeliverable 
software  is  obtained  from  the  government  or  a 
commercial  source,  it  has  to  be  O.K. 
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Technical  Tasks  in  Software  Acquisition 

What  technical  tasks  must  the  System  Program  Office 
(SPO)  and  in-plant  Contract  Administration  Service  (CAS) 
organizations  perform  in  the  acquisition  of  mission  critical 
computer  resources  (MCCR)  software? 

Tasks  in  software  acquisition  were  selected  on  the 
basis  of  these  criteria: 

(1)  Task  should  represent  an  appreciable  volume  of 
work  to  be  done,  usually  by  more  than  one 
individual . 

(2)  Task  should  represent  a  process  rather  than  an 
event . 

(3)  Task  should  be  software  dominated. 

(4)  Task  should,  as  much  as  possible,  represent  a 
description  of  "what  is  to  be  done"  rather  than 
"how  to,"  or  criteria,  or  policy. 

A  total  of  89  technical  tasks  were  identified  and  are 
displayed  in  detail  in  Table  Three  of  the  text.  While  Table 
Three  is  intended  to  be  comprehensive,  there  are  some 
restrictions  which  must  be  made  in  this  sense.  First,  there 
is  the  problem  of  diversity.  Although  one  criterion  in 
selecting  tasks  for  this  analysis  was  concreteness,  this 
criterion  itself  (though  useful)  mitigates  against 
comprehensiveness.  Software  applications  in  embedded 
systems  are  so  diverse  that  any  given  project  may  have  a 
great  deal  of  uniqueness  in  its  software.  For  this  reason, 
the  only  way  to  cover  all  possibilities  is  to  present  tasks 
that  are  extremely  generic  or  abstract.  Hopefully,  a  useful 
middle  ground  has  been  achieved  here. 
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However,  any  given  application  may  certainly  require 
tasks  not  represented  here. 

The  second  problem  for  comprehensiveness  involves  the 
problems  of  the  extent  of  the  effects  of  software.  With 
increasing  applications  of  computers  and  complexity  of 
software,  it  is  difficult  to  delimit  the  extent  of  the 
effects  of  software.  The  selection  of  tasks  for  this  study 
implies  a  delimitation  of  the  systemic  influences  of 
software.  It  is  noted  that  this  is  sensible  but  arbitrary. 

Relation  of  Tasks  to  SPO  and  CAS 

The  separation  of  responsibilities  for  technical  tasks 
(shown  in  Table  Five  of  the  text)  between  SPO  and  CAS 
contains  a  great  deal  of  information.  However,  the  general 
implications  of  this  separation  can  be  characterized  as 
reflecting: 

(1)  An  increased  amount  of  required  consultation  of 

the  CAS  by  the  SPO  than  is  currently  reflected  in 
written  policy  and/or  practice.  This  is 
especially  true  in  the  areas  of  pre-award  survey 
and  RFP  development.  Interviews  conducted  in  the 
study  indicate  that  under  current  practice, 
valuable  knowledge  developed  by  CAS  agencies 
through  their  long-term  relationships  with  day-to- 
day  contractor  activities  is  under  utilized  and 
wasted  in  these  areas.  If  followed,  the 
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recommendations  of  Table  Five  should  alleviate 
this  situation,  which  should  not  be  allowed  to 
continue  in  the  critical,  costly,  and  problem- 
ridden  area  of  software  acquisition. 

(2)  More  compulsory  participation  of  CAS  personnel  in 
formal  reviews  during  the  development  process.  As 
specified  here,  required  CAS  participation  would 
begin  with  the  System  Design  Review  (Table  Three, 
number  66) .  Such  early  involvement  is  necessary 
in  order  for  CAS  personnel  to  do  their  jobs 
effectively  through  the  later  stages  of  the 
software  development  process.  Also,  CAS 
familiarity  with  day-to-day  problems  of  software 
development  offers  the  potential  to  predict  and 
prevent  potential  problems  earlier  in  the 
development  process.  This  offers  the  possibility 
of  great  dollar  and  time/schedule  savings  to  the 
government . 

CAS  Personnel  Skills 

The  tasks  identified  in  this  study  suggest  the 
designation  of  three  main  areas  of  training/knowledge  for 
CAS  personnel: 

(1)  A  core  area,  referred  to  as  Software  Development 

Management  Evaluation.  This  area  is  introductory, 
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foundational,  and  basic,  and  recommended  for  all 
affected  CAS  personnel. 

(2)  An  area  specific  to  Engineering,  referred  to  as 
Software  Development  Technical  Engineering 

Evaluation. 

(3)  An  area  specific  to  Quality  Assurance,  referred  to 
as  Software  Quality  Assurance  Evaluation. 

More  specific  areas  for  each  main  heading  above  are 
outlined  in  the  text. 

A  specific  recommendation  is  made  that  further 
development  of  training  proceed  in  the  direction  of 
establishing  systems  of  training  objectives  similar  to  the 
system  of  KSAs  (Knowledges,  Skills,  and  Abilities)  currently 
used  by  Quality  Assurance.  Such  systems  of  specific 
training  objectives  should  lend  to  more  effective  and  more 
cost-effective  training,  in  addition  to  other  benefits. 

Further  needed  increases  in  manpower  acquisition  for 
CAS  software  support,  estimated  by  the  Electronics  Institute 
of  America,  average  14%  per  year  for  the  next  three  years. 
Currently,  the  effects  of  Gramm-Rudman  have  not  seriously 
eroded  numbers  of  software  CAS  personnel.  However,  given 
that  there  was  already  a  shortage  of  well-trained  software 
personnel  and  a  needed  increase  of  14%  per  year,  it  is  clear 
that  holding  the  status  quo  amounts  to  losing  ground. 
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Policy  Rp-coTnmendations 

The  task-relations  specified  in  Table  Five  are 
designated  as  recommendations  for  policy,  and  it  is 
recommended  that  policy  should  be  altered  to  enforce  the 
relationships  of  Table  Five.  This  contractor  is  aware  that 
many  of  the  relationships  shown  in  Table  Five  already  exist 
in  written  policy  as  "suggested,"  "where  appropriate"  (or 
similar  phrases)  or  are  even  required.  However,  interviews 
conducted  in  this  study  show  that  in  spite  of  guidance  or 
requirements  of  written  policy,  in  practice,  there  is 
insufficient  involvement  of  CAS  capabilities  in  the 
development  of  RFPs  and  contracts  and  too  little  early 
involvement  of  CAS  personnel  in  formal  reviews  and  audits  to 
fully  utilize  and  support  their  potential.  Therefore, 
policy  should  be  rewritten  to  require  and  enforce  CAS 
involvement  as  shown  in  Table  Five  and  discussed  in  Chapter 
6.0  and  8.0  of  this  report. 

Ancillary  Recommendations 

Based  on  informal  observations  and  considerations,  two 
additional  recommendations  are  offered. 

The  first  involves  a  "services-marketing"  approach  for 
CAS.  This  is  based  on  the  idea  of  not  relying  solely  on  the 
rearrangement  and  enforcement  of  policy  to  attain  increased 
involvement  for  CAS  in  preaward  activities  and  greater  early 
contract  involvement.  The  idea  is  to  package  these  CAS 
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services  in  terms  of  benefits  to  the  SPOs  and  aggressively 
and  persuasively  market  them. 

The  second  involves  streamlining  of  training.  This 
could  be  done  by  establishing  an  entry  or  core  course  for 
all  CAS  personnel  (horizontal  and  vertical)  involved  with 
software.  More  advanced  and  specific  training  could  then  be 
placed  in  a  dynamic  data  base  which  could  be  continuously 
expanded  and  updated.  This  data  base  could  be  accessed  on 
an  ad  lib  basis  by  individuals  and  groups.  This  would 
provide  highly  tailored  and  individualized  training  at  low 
dollar  and  time  cost. 
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1.0  GENERAL  DIFFERENCES  IN  HARDWARE  AND  SOFTWARE 


1.1  Introduction . 

Historically,  weapons  systems  have  been  dominated  by 
hardware  orientations.  Until  very  recent  times,  weapons 
were  essentially  hardware  systems  operated  entirely  by 
humans.  Although  the  weapons  themselves  could  be  adjusted, 
these  adjustments  were  determined  and  carried  out  (mostly) 
by  fairly  direct  human  judgment  and  action.  With  the  advent 
of  computers  in  weapons  systems,  many  adaptive  procedures 
and  criteria  could  be  programmed  into  systems  via  software 
which  extended  and  removed  (in  time  and  space)  the  effects 
of  human  operators  in  many  cases.  This  essentially  amounts 
to  turning  over  a  certain  amount  of  control  of  a  weapons 
system  to  computer  software. 

For  historical  reasons,  hardware-oriented  methods  and 
approaches  to  weapons  system  acquisition  predominate. 
Currently,  control  aspects  of  these  systems  is  coming  to  be 
dominated  by  software,  and  the  appropriate  methods  of 
approaches  for  the  acquisition  of  weapons  system  software 
bear  subtle  and  critical  differences  from  parallel  methods 
for  hardware.  A  reflection  of  the  increasing  amount  of 
software  involvement  in  one  organization  is  shown  in  Figure 
One.  (The  directness  of  the  relationship  between 


Percent  of  Development  Personnel 


Dennis  Blosser  &  Associates 
Final  Report 
Page  2 


FIG.  ONE 


GROWTH  IN  SOFTWARE 
PERSONNEL 
(Glaseman,l982) 
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percentage  of  personnel  and  amount  of  software  produced  is 
clearly  an  open  question.) 

Chapters  1.0  through  4.0  cover  the  preliminary  phase 
(Phase  II)  of  an  investigation  into  the  differences  in 
hardware  and  software  for  contract  administration,  how  these 
differences  impact  contract  administration  technical 
functions,  training,  needs  and  staffing  needs,  and 
implications  for  administration  of  nondeliverable  computer 
software.  The  observations  of  these  chapters  serve  the 
purpose  of  laying  groundwork  for  more  definitive 
observations  in  Chapter  5.0  through  9.0. 

1.2  Method 

Investigation  for  the  preliminary  part  of  the  study 
consisted  of  document  study  and  interviews.  Its  general 
goal  was  to  give  the  contractor  first-hand  contact  with  some 
of  the  agencies  involved  in  order  to  provide  a  realistic 
foundation  for  issues  to  be  examined  more  formally  in  the 
final  portion  of  the  study.  In  addition  to  the  examination 
of  documents  such  as: 

*  DOD-STD-2167 

*  MIL-S-52779A 

*  MIL-STD-1679 

*  DOD-STD-2168  (May  9,  1985,  draft  version) 

*  AFCMD  Regulation  800-3 

* 


AFSCR  800-42 
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*  AFCMD  PAMPHLET  800-2 

*  AFCMDR  74-1 

*  DOD-STD-480A 

and  other  government  guides,  etc.,  the  contractor  examined 
Statements  of  Work  and  Requests  for  Proposals.  The 
contractor  also  gained  valuable  information  and  guidance 
from  on-site  meetings  with  personnel  at  the  following 
agencies: 

*  AFCMD/ HQ  Kirtland  AFB,  NM 

*  SD/SPV  Space  Division,  Los  Angeles,  CA 

*  AFPRO/Martin-Marrietta ,  Denver,  CO 

The  observations  begin  with  a  general  discussion  of 
differences  in  software  and  hardware.  The  subtle 
similarities  and  differences  in  the  engineering  of  software 
and  hardware  must  be  clearly  differentiated.  Insistence  on 
detailing  these  differences  does  not  imply  that  either 
government  technical  assistance  personnel  or  DOD  contractors 
are  ignorant  of  them.  More  sophisticated  members  of  these 
groups  are  well  aware  of  these  differences,  and  less 
sophisticated  members  no  doubt  easily  recognize  these 
differences  when  they  are  pointed  out.  Nevertheless,  these 
differences  are  worth  noting,  because  in  one  form  or 
another,  they  continue  repeatedly  to  be  the  underlying 
causes  of  the  unique  and  serious  problems  that  arise  in  the 
development  of  large  software  projects.  Also,  although 
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differences  in  hardware  and  software  are  often  elsewhere 
referred  to  by  implication  or  in  a  passing  manner,  they  are 
seldom  dealt  with  directly  or  in  detail.  Failure  to  make 
these  differences  distinct  has  already  contributed  to  a 
great  deal  of  confusion  in  the  development  of  software.  The 
confusion  will  be  compounded  unless  these  distinctions  are 
clearly  highlighted. 

Following  the  discussion  of  the  differences  in  software 
and  hardware,  the  critical  differences  for  contract 
administration  in  life  cycle  are  discussed.  This  is 
followed  by  possible  implications  for  staffing  and 
organization  and  a  brief  discussion  of  implications  for 
nondeliverable  software. 

1.3  Hardware  vs.  Software  —  Nine  Important  Differences. 

Many  of  the  differences  between  hardware  and  software 
that  are  outlined  and  discussed  here  may  seem  obvious  to 
those  with  a  good  deal  of  experience  in  these  matters. 

Still,  these  differences  are  worth  discussing  for  several 
reasons: • 

(1)  Although  many  who  work  for  the  government  in  MCCR 
software  are  quite  sophisticated,  many  others  are 
definitely  not  so  sophisticated, 

Although  the  differences  discussed  here  may  have 
been  noted  by  others,  they  are  usually  given 
partial,  limited,  or  elliptical  treatment, 


(2) 
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(3)  Currently,  neither  government  nor  the  commercial 
sector  engineers  and  manages  the  development  of 
MCCR  software  with  satisfactory  effectiveness. 
Virtually  all  of  the  critical  problems  of  software 
development  arise  from  or  are  profoundly  affected 
by  the  basic  differences  between  hardware  and 
software  (Fox,  1982) .  Thus,  until  we  can  say  we 
have  significantly  controlled  the  problems  of 
software  development,  it  is  worthwhile  to  continue 
to  consciously  re-examine  these  matters  both  for 
the  sophisticated  and  the  not-so-sophisticated. 

MCCR  software  is  part  of  an  engineered  weapons  system. 
As  such,  this  software  itself  must  be  engineered  and 
correspond  to  the  life-cycle  development  process  of  weapons 
systems.  The  application  of  life-cycle  concepts  and 
engineering  principles  to  software  marked  a  great  advance  in 
the  history  of  computer  applications.  However,  there 
appears  to  be  a  strong  organizational  tendency  to  exert  a 
hardware-engineering  mind-set  when  applying  engineering 
principles  to  software.  This  mind-set  overlooks  the 
profound  differences  between  hardware  and  software. 

Although  the  engineering  principles  applicable  to 
software  and  hardware  are  similar,  the  tremendous 
differences  in  the  essential  nature  of  the  two  products 
requires  that  the  methods  for  applying  these  principles  be 
different.  Since  these  differences  and  organizational 
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tendencies  to  deal  with  them  less  effectively  than  desired 
form  the  basis  for  this  study,  the  differences  are  discussed 
here  in  some  detail.  Table  One  shows  the  nine  areas  of 
software  versus  hardware  similarities  and  differences  which 


will  be  discussed. 


1.3.1 
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Physical  Existence.  Hardware  has  physical 
existence.  It  consists  of  natter  and  takes  up  space. 
Software,  by  contrast,  does  not  enjoy  physical  existence  in 
that  sense.  The  nonphysical  existence  of  software  is 
sometimes  termed  "virtual"  as  opposed  to  physical. 

Similarly,  sometimes  software  is  referred  to  as  information 
or  data  consisting  of  instructions  for  a  computer.  Often, 
we  casually  speak  of  a  computer  program  (software)  as  an 
object.  However,  the  closest  software  comes  to  being  an 
object  is  actually  a  representation  of  that  software  in  some 
medium  (e.g.  print,  magnetic  disk) .  These  may  seem  like 
fine  distinctions,  but  they  are  legitimate,  and  they  turn 
out  to  be  crucial.  Most,  if  not  all,  of  the  problems  unique 
to  developing  MCCR  software  arise  from  its  nonphysical 
property.  Consider  any  traditional  field  of  engineering. 
Consider  the  various  fundamentals  of  measurement, 
observation,  modeling,  testing,  etc.,  in  that  field  of 
engineering.  Consider  how  those  fundamentals  could  (or 
could  not)  be  applied  to  a  component  that  is  invisible, 
intangible,  and  has  no  physical  existence. 


Dennis  Blosser  &  Associates 
Final  Report 
Page  10 


************************* 

*  Application  Note  for  In-plant  Technical  * 

*  Assistance:  Observation  of  the  software  as  * 

*  it  develops  (at  the  module  level,  say)  is  * 

*  always  indirect.  This  means  AFPRO  personnel  * 

*  may  be  looking  at  code,  or  being  given  an  * 

*  electronic  representation,  or  looking  at  * 

*  plans,  or  specifications,  or  test  results.  * 

*  Making  a  useful  choice  among  these  and  inter-  * 

*  preting  them  requires  different  knowledge  and  * 

*  criteria  than  would  be  applicable  for  compar-  * 

*  able  observations  of  hardware  development.  * 

*  Without  adequate  orientation  and  training,  * 

*  AFPRO  personnel  may  feel  that  software  doesn't* 

*  give  them  anything  to  "get  a  handle  on”  and  * 

*  thus  avoid  software  duties  when  possible.  * 

************************* 


1.3.2  Physical  Dimensions.  Hardware  has  physical 
dimensions,  the  simplest  being  those  such  as  length,  width, 
mass.  Since  software  lacks  physical  existence,  its 
dimensions,  similarly,  are  less  concrete.  The  tangible 
physical  dimensions  of  hardware  form  the  basis  for  some  very 
realistic  guidelines  and  limits  in  the  design  and 
development  of  systems.  For  example,  if  a  system  component 
is  proposed  to  a  mechanical  engineer  which  requires  a  large 
number  (say  500,000)  of  moving  parts,  he  will  probably 
quickly  reject  its  feasibility.  It  is  too  large  in  number 
of  moving  parts  —  it  just  wouldn't  work.  However,  the  size 
of  software  systems  seems  to  be  limited  only  by  computers 
with  memories  large  enough  to  execute  them  and  the 
imagination  of  designers.  Because  of  concrete  physical 
dimensionality,  a  too-large  hardware  system  becomes 
obviously  untrustworthy  at  a  relatively  small  level. 
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Lacking  physical  dimensionality,  there  is  no  clear,  simple 
guide  in  software  as  to  how  big  is  too  big,  as  long  as  we 
think  we  understand  it  and  have  a  big  enough  computer  to  run 
it.  [Not  too  long  ago,  we  would  have  thought  that  physical 
and  memory  size  of  computers  would  soon  provide  clear-cut 
limits  for  size  of  software  systems,  but  current  trends  are 
just  the  opposite.  Vastly  more  and  more  computer  (hardware) 
power  available  at  less  and  less  cost  in  dollars,  time, 
space,  and  weight  is  pushing  software  in  the  direction  of 
larger  and  larger  systems . ]  Some  attempts  are  underway  at 
developing  a  software  "physics,"  but  these  are  at  a  very 
early  and  tentative  level,  and  their  results  are  not 
available  or  appropriate  for  immediate  engineering 
applications.  In  all  likelihood,  developments  in  these 
areas  will  proceed  slowly  due  to  their  academic  and 
theoretical  nature. 

************************* 

*  Application  Note  for  In-Plant  Technical  * 

*  Assistance;  About  the  only  practical  control  * 

*  AFPRO  personnel  can  exert  on  size  (and  thus  * 

*  complexity)  in  software  in  MCCR  applications  * 

*  is  at  the  module  level.  The  trend  is  clearly  * 

*  toward  larger  and  larger  software  systems.  * 

*  So,  the  best  control  for  size  is  to  keep  mod-  * 

*  ules  to  a  size  limit.  A  common  rule  of  thumb  * 

*  is  that  a  module  should  be  less  than  100  lines* 

*  of  code  on  the  average  and  should  not  exceed  * 

*  200  lines  of  code  at  the  extreme.  A  better  * 

*  approach  would  be  to  design  modules  that  ex-  * 

*  hibit  loose  coupling  (i.e.,  relative  independ-* 

*  ence)  to  other  modules  and  that  exhibit  inter-* 

*  nal  cohesiveness  or  high  internal  relation-  * 

*  ships  and  then  post-apply  the  numeric  rules  of* 

*  thumb  as  contraints.  In  any  case,  the  appro-  * 

*  priate  control  point  for  AFPRO  surveillance  is* 

*  the  Unit  Development  Folder/Module  level.  * 

************************* 
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1.3.3  Measurability.  Due  to  its  physical  dimensionality, 
hardware  is  directly  measurable.  This  is  not  to  say  that 
direct  measurements  are  the  only  useful  measurements  for 
hardware.  However,  if  it  weren't  for  the  feasibility  of 
precise,  reproducible  measurements,  we  could  never  have 
produced  hardware  whose  functions  would  be  interesting  to 
measure.  Indeed,  it  has  often  been  argued  that  measurement 
is  the  fundamental  basis  of  science  and  the  fundamental 
basis  of  engineering. 

Software,  having  no  physical  dimensions,  offers 
difficulties  for  the  proposition  of  measurement.  Consider, 
for  instance,  the  simple  question  "How  large  is  Program  A?" 

A  very  simple  answer  would  be  to  state  how  long  Program  A  is 
in  lines  of  code.  But  this  leaves  open  questions  such  as: 

Is  the  code  assembly  language,  or  a  high-order  language? 
Which  high-order  language?  Are  all  programs  with  the  same 
number  of  lines  of  code  in  the  same  language  truly  of  equal 
size  [No.]?  And  so  on. 

These  measurement  difficulties,  in  turn,  pose 
difficulties  for  AFPRO  personnel  in  just  about  all  areas  of 
contract  management  regarding  software.  Evaluation  of  the 
propriety  of  software  analysis,  software  design,  and 
software  implementation  methods  is  made  more  difficult  by 
the  measurement  problem.  Progress  in  relation  to  cost  and 
schedule  expectations  is  difficult  to  evaluate.  The  various 
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factors  constituting  final  product  quality  in  software  are 
difficult  to  evaluate. 

The  measurement  of  software  is  being  studied 
intensively,  and  developments  are  occurring  at  a  rapid  pace. 
However,  there  are  as  yet  no  thoroughly  quantifiable  and 
satisfactory  answers  to  many  of  the  basic  needs  of  AFPROs  in 
the  area  of  measurement.  At  the  conceptual  level,  good 
clarifications  have  been  made  of  some  of  the  basic  factors 
which  are  desireable  to  measure  in  software.  However,  these 
factors  are  functional  (e.g.  understandability ,  testability) 
and  as  yet  very  few  can  be  quantified  to  any  degree,  if  at 
all.  As  more  and  more  satisfactorily  quantifiable  measures 
of  software  become  available,  software  engineering  will 
benefit  greatly  and  mature.  This  will  lead  to  better 
contractor  capability  and  an  increase  in  the  ability  of 
AFPROs  to  perform  critical  surveillance.  It  should  be 
clear,  however,  that  even  with  such  maturation,  the  basic 
differences  between  software  and  hardware  will  make  the 
methods  of  measurement  different,  although  the  principle  of 
measurement  spans  both  areas. 
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************************* 

*  Application  Note  for  In-Plant  Technical  * 

*  Assistance:  Currently,  AFPRO  personnel  could  * 

*  conceivably  choose  among  metrics  which  are  * 

*  numerically  quantifiable  and  metrics  which  are* 

*  conceptual,  but  not  numerically  quantifiable  * 

*  (or  in  some  cases  numeric,  but  questionable  to* 

*  interpret) .  There  are  metrics  for  software  * 

*  development  process,  metrics  for  software  * 

*  development  output,  and  some  which  are  not  * 

*  clear  cut.  There  are  also  measures  available  * 

*  which  are  useful  management  indicators  which  * 

*  are  not  technically  software  metrics.  The  * 

*  best  pay-off  is  to  identify  the  decisions  * 

*  critical  for  MCCR  contract  management  (and  at  * 

*  what  state  in  the  development  process)  and  * 

*  then  search  for  a  combination  of  measures  (in-* 

*  dicators  and/or  metrics)  which  are  affordable  * 

*  to  collect  and  compute  and  which  can  be  use-  * 

*  fully  interpreted,  possibly,  in  combination.  * 

*  AFPRO  personnel  should  be  aware  that  metrics  * 

*  and  indicators  with  low  collection/computation* 

*  cost  and  high  interpretation/decision  value  * 

*  may  not  exist  for  all  desireable  situations.  * 
************************* 


1.3.4  Standard  Elements.  Many  of  the  elements  of  hardware 
engineering  are  highly  standardized.  Their  make-up  and 
functions  are  transportable  with  high  reliability  and 
accuracy,  and  much  is  known  about  their  behaviors  and 
limits.  Furthermore,  this  standardization  spans  a  very  wide 
scope.  For  example,  it  can  include  single,  elementary  units 
of  electronic  components,  such  as  transistors,  that  are 
extremely  small  and  separate,  up  to  complex  electro¬ 
mechanical  devices  whose  operation  is  considered  a  fully 
standardized  procedure  under  proper  conditions  of 
calibration  and  usage. 
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By  comparison,  the  current  state  of  standardization  for 
software  is  almost  the  opposite  (this  is  in  no  way  meant  to 
demean  efforts  such  as  STARS  and  Ada,  but  to  emphasize  a 
situation  that  remains  seriously  problematic  in  spite  of 
those  admirable  efforts) .  There  are  many  different  major 
languages  used  in  MCCR  by  DOD,  and  even  many  of  these 
languages  have  different  dialects  and  versions  which  differ 
depending  on  the  hardware  system  they  utilize  (and  there  are 
very  many  different  hardware  systems) .  Some  studies  have 
counted  the  number  of  languages  and  variants  utilized  within 
DOD  to  run  literally  into  the  hundreds.  It  seems  hard  to 
believe,  but  counts  notwithstanding,  there  is  a  definite  and 
pernicious  proliferation  of  languages  in  MCCR  software. 

Grady  Booch  (1983)  has  enumerated  several  of  the  costs 
linked  to  proliferation  of  languages  in  the  DOD  such  as: 
dilution  of  effects  of  training,  almost  no  technology 
transfer  among  projects,  and  a  general  diffusion  of 
resources.  A  number  of  studies  of  the  proliferation  of 
computer  languages  in  DOD-embedded  systems  pointed  to  the 
desirability  of  a  single  standard  language  and  the  issuance 
of  Directives  5000.29  and  5000.31  which  served  to  narrow  the 
range  of  acceptable  languages. 

Nevertheless,  the  effects  of  these  studies,  the  advent 
of  Ada,  and  directives  do  not  as  yet  appear  to  have  deeply 
affected  contractor  software  development  practices. 
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The  result  of  this  is  that  relatively  few  standard 
elements  or  systems  of  software  exist.  Even  the  fundamental 
elemental  statements  of  a  computer  language  (e.g.,  FORTRAN) 
may  not  execute  identically  in  different,  but  functionally 
similar  environments.  While  this  lack  of  standardization 
may  not  represent  an  intrinsic  property  of  software  as 
contrasted  to  hardware,  it  is  currently  and  will  continue  to 
be,  influenced  by  the  peculiar  nature  of  software. 

1.3.5  Life-Cycle  Relationships.  Hardware  engineering  and 
the  engineering  of  modern  weapon  systems  gave  rise  to  the 
Life-Cycle  concept  of  weapons  development.  The  Life-Cycle 
concept  has  provided  great  advantages  in  the  successful 
development  of  systems,  the  management  of  resources 
throughout  the  Life-Cycle,  assurance  of  conformation  to 
standards,  and  other  desireable  achievements.  The  Life- 
Cycle  concept  initially  evolved  around  systems  that  were  not 
at  all  computerized.  So,  the  mind-set  associated  with  the 
Life-Cycle  originated  in,  and  has  a  long  history  of,  purely 
hardware  orientation.  When  computers  began  to  be  introduced 
into  weapons  systems,  there  was  certainly  not  much  initial 
motivation  to  view  the  situation  as  being  different  in  any 
important  sense.  Although  computers  were  novel,  and  their 
functions  were  novel,  computers  are  hardware.  There  was 
already  a  well-established  history  of  experience  with 
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complex  electronic  hardware,  and  the  computer,  as  hardware, 
fit  quite  well  into  that  mind-set. 

Also,  and  this  is  the  most  critical  point .  the  early 
computers  were  small  (operating  capacity)  and  weak. 
Correspondingly,  the  software  which  controlled  those 
computers  was  small  and  easily  managed.  It  seemed  quite 
appropriate  under  those  circumstances  to  continue  with  the 
hardware  Life-Cycle  concept  for  the  system,  including  the 
computer  hardware,  and  simply  require  the  software  as  a  data 
item  on  contracts,  much  like  documentation  for  operation  of 
the  computer  hardware.  For  that  time,  that  thinking  was 
quite  appropriate.  It  would  still  be  appropriate  except 
that  some  sweeping  and  profound  changes  have  occurred. 

In  the  brief  thirty  or  so  years  that  have  elapsed  since 
digital  computers  first  began  to  be  integrated  into  weapons 
systems,  their  functions  have  increased  tremendously  in  two 
important  ways.  First,  computers  have  come  to  operate  in  a 
greater  and  greater  diversity  of  roles.  Early  on,  computers 
were  used  in  only  a  few,  isolated  applications  in  weapons 
systems.  Now,  computer  applications  in  weapons  systems  have 
multiplied  to  the  point  of  seeming  to  be  everywhere.  As  an 
example,  in  early  applications,  a  very  large  weapon  system 
such  as  a  bomber  aircraft  might  have  one  or  two  on-board 
computers  to  assist  in  such  things  as  sighting  for  bomb- 
drops  or  navigation.  Currently,  a  smaller  system,  such  as  a 
fighter  aircraft,  has  multiple  on-board  computers  that 
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control  many  functions  of  the  aircraft  much  like  a 
biological  nervous  system.  There  are  computers  to  manage 
the  aiming  and  firing  of  various  weapons  on  board,  and  some 
weapons  (e.g.,  missiles)  are  under  articulated  computer 
guidance  after  firing.  (Even  this  description  greatly 
simplifies  and  understates  the  involvement  of  computers  in 
this  example.) 

The  second  important  change  that  is  linked  to  increased 
diversity  is  increasingly  complex  function.  The  almost 
unbelievable  advances  in  microminiaturization  of  computer 
circuitry  have  supported  more  different  applications  by 
providing  more  computer  power  (complexity  of  function)  per 
dollar  and  per  unit  of  weight  and  space  than  was  dreamed  of 
thirty  years  ago. 

Given  the  peculiar  nature  of  software  and  the  problems 
of  engineering  software  as  outlined  here,  the  results  for 
scientific  and/or  engineering  development  of  software  have 
been  virtually  disastrous.  Obviously,  the  function  of 
computers  in  weapons  systems  is  to  control  or  assist  in 
control  of  a  system  or  its  components.  But,  and  this  is 
critical .  the  computers  themselves  are  in  turn  controlled  by 
software .  So  here  is  the  crisis:  while  the  potential  for 
control  in  weapons  systems  through  advances  in  computer 
hardware  has  grown  like  a  super-exponential,  the  realization 
of  that  potential  is  severely  hindered,  jeopardized,  and 
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made  expensive  by  a  definite  lack  of  similar  advancements  in 
the  software  required  to  control  the  computers. 

When  it  first  became  clear  (circa  1968)  that  software 
development  was  getting  out  of  control  to  the  point  that  the 
idea  of  "engineering"  software  began  to  emerge,  the  Life- 
Cycle  concept  began  to  be  applied  to  software.  Like  the 
other  concepts  discussed  here,  in  principle .  this  is 
appropriate  and  valuable.  Problems  arise,  however,  if  a 
hardware  mind-set  is  maintained  in  applying  the  Life-Cycle 
principle  to  software.  This  is  like  the  case  of  the  other 
issues  discussed  here.  The  principles  are  the  same,  but  due 
to  the  basic  differences  in  hardware  and  software,  the 
applications  must  be  different  in  order  to  effective. 

The  Life-Cycle  concept  as  a  principle  applies 
fruitfully  to  software.  Problems  arise  when,  in  the  context 
of  a  hardware  weapons  system,  the  subtle  differences  in  the 
meaning  of  the  concept  of  Life-Cycle  for  software,  and 
specifically  how  the  software  Life-Cycle  is  integrated  into 
the  overall  weapons  system  Life-Cycle,  are  neglected. 

A  detailing  of  how  some  of  these  problems  can  arise  in 
DOD  contract  administration  appears  in  Section  2.2. 

However,  it  is  likely  that  the  implications  of  Life-Cycle 
differences  in  hardware  and  software  are  likely  to  be 
neglected  where: 

(1)  The  history  of  the  organization  is  heavily 
grounded  in  hardware  systems . 
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(2)  The  organization's  concepts  of  Life-Cycle 
originated  in  hardware  and  has  a  long  history 
where  hardware  orientation  predominates. 

(3)  Few  of  the  organization's  front-line  workers  have 
extensive,  formal  training  in  software,  but  do 
have  extensive  training/experience  with  hardware. 

(4)  Practices  of  hiring,  promotion,  retention,  and 
compensation  mitigate  against  rapid  adjustment  to 
new  technology. 

Obviously,  an  organization  that  meets  the  above  requirements 
is  DOD. 

This  is  not  to  say  that  there  have  not  been  definite 
and  appropriate  attempts  made  to  provide  DOD  workers  with 
conceptual  and  legal  tools  to  work  with  software.  Simply, 
given  the  organizational  nature  of  DOD,  the  historical  DOD 
hardware  mind-set,  the  problems  facing  the  software  industry 
in  general,  and  the  infant  nature  of  software  as  compared  to 
the  demands  of  hardware  potential,  efforts  have  fallen 
short. 

!*3.6  Design  Precision.  Creativity  and  perhaps  even  some 
degree  of  artistry  are  no  doubt  valuable  abilities  in  any 
design  undertaking.  But,  the  orientation  of  the  hardware 
system  design  is  to  always  provide  as  rational  and 
quantifiable  a  means  as  possible  for  evaluating  different 
alternatives  of  design.  Of  course,  not  even  in  hardware  can 
all  design  decisions  be  suitably  or  profitably  quantified. 
But,  the  limitations  for  quantitative  evaluation  of  designs 
in  software  is  much  more  limited  than  in  hardware.  These 
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limitations  go  back  to  the  non-physical  nature  of  software 
and  current  problems  of  software  measurement.  There  are 
definitely  methodologies  available  in  software  design.  And, 
they  offer  definite  rationales  and  advantages.  However, 
software  designs  are  ultimately  like  very  precise  verbal 
statements.  Hardware  designs,  on  the  other  hand,  are  like 
very  precise  descriptions  of  objects,  measurements,  and 
actions.  One  way  of  comparison  is  to  consider  the  use  of 
diagrams  in  hardware  vs.  software.  Design  diagrams  in 
hardware  ultimately  are  a  way  of  creating  images 
representing  physical  objects.  Design  diagrams  in  software 
are  a  way  of  creating  images  which  represent  data,  or 
information,  or  commands — non-physical  entities.  In  final 
analysis,  design  of  software  is  more  abstract  than  the 

ar 

design  of  hardware.  As  an  abstraction,  it  lends  itself  to 
more  possible  instantiations  than  does  a  design  for 
hardware.  Thus,  it  is  essentially  less  precise.  Some 
writers  (e.g.,  Martin,  1985;  Hoare,  1975)  have  argued  that 
the  practice  of  software  design  and  development  lacks  so 
much  precision  in  areas  such  as  cost  control,  reliability, 
and  control  of  complexity  that  it  scarcely  dares  to  be 
associated  with  the  term  "engineering"  as  it  is  commonly 
understood. 

1.3.7  Cost  of  Design  Correction.  In  many  hardware 
components,  minor,  sometimes  even  relatively  major,  design 
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alterations  can  be  reasonably  allowed  during  production. 
Thus,  a  component  may  meet  minimal  acceptance  standards  and 
yet  receive  profitable  alteration  at  a  later  phase.  In 
other  words,  some  changes  in  hardware  design  exhibit 
beneficial  cost-effectiveness  well  after  the  design  phase. 

By  contrast,  alterations  in  software  systems,  even 
apparently  minor  ones,  after  the  design  phase  become 
tremendously  expensive.  So  expensive,  in  fact,  that  a  rule 
of  dollar  and  time  economy  is  to  make  as  many  required 
changes  as  can  be  anticipated  before  leaving  the  design 
phase.  Thus,  relative  to  hardware,  the  design  phase  should 
be  intensive  and  highly  test-oriented.  The  primary  reason 
for  these  differences  has  to  do  with  the  quality  of 
modularity.  Most  components  of  hardware  systems  are  highly 
functionally  encapsulated.  They  affect  each  other  only 
through  clearly  specified  and  controlled  interfaces. 
Currently,  the  components  of  a  software  system  seldom 
exhibit  strong  modularity.  That  is,  they  link  to  other 
components  not  only  via  their  designed  interfaces,  but  often 
through  unintentional  and/or  unpredictable  links  to  other 
components.  Unwanted  effects  going  from  one  software 
component  to  another  in  this  fashion  are  called  side 
effects . 

Thus,  when  highly  modular  hardware  components  are 
redesigned  during  post-design  phases,  their  system-wide 
effects  are  highly  predictable.  In  the  case  of  software. 


Dennis  Blosser  &  Associates 
Final  Report 
Page  23 


this  predictability  does  not  hold.  Due  to  weak  modularity, 
redesign  of  a  software  system  component  in  post-design  phase 
may  have  wide  ranging  side  effects  which  will  then  require 
affected  components  to  need  redesign,  which  may  in  turn  have 
wide  ranging  side  effects,  which  will  require  redesign  of 
other  components,  etc.  The  problem  of  modularity  is  well 
noted  but  by  no  means  conquered.  At  present,  the  best 
solutions  appear  to  be  to  anticipate  and  solve  as  many 
problems  as  possible  and  to  try  to  build  in  modularity  in 
the  design  stage  —  easy  to  state,  difficult  to  accomplish. 

************************* 

*  Application  Note  for  In-plant  Technical  * 

*  Assistance ;  In  practice,  in  spite  of  philoso-* 

*  phies  of  "early  involvement,"  AFPRO  surveil-  * 

*  lance  is  most  heavily  oriented  to  testing,  * 

*  with  little  or  virtually  no  strong  involvement* 

*  in  design  phases.  However,  the  capability  for* 

*  the  program  to  be  effective  in  the  test  phases* 

*  must  be  laid  down  and  assured  from  the  earli-  * 

*  est  design  phases.  Thus,  not  only  is  it  cost  * 

*  effective  to  correct  problems  in  the  design  * 

*  phase,  when  feasible,  but  it  is  also  neces-  * 

*  sary  to  assure  the  quality  of  testability  from* 

*  the  earliest  design  phases.  Otherwise,  the  * 

*  test  phase  can  be  explosive  in  terms  of  cost,  * 

*  schedule,  and  morale  (Evans,  1984) .  * 

************************* 


1.3.8  Quality  Assurance.  In  simplified  terms  quality 
assurance  is  concerned  (1)  that  quality  be  designed  into  the 
product,  (2)  that  quality  be  designed  into  the  production 
process  and  (3)  that  quality  be  executed  in  the  production 
process  as  evidenced  by  evaluation  of  end-products.  In 
spite  of  any  theoretical  positions  or  current  management 
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philosophies,  the  orientation  of  DOD  to  quality  assurance  is 
understandably  hardware-dominated,  and  as  such,  places  its 
greatest  manpower  effort  in  the  third  area  of  quality 

described  above.  In  the  case  of-hardware,  this  is 
probably  a  good  strategy.  In  pre-production  and  production 
development,  the  government  definitely  participates,  but 
puts  the  large  burden  of  technological  manpower  on  the 
contractor.  But,  in  the  final  phase  of  quality  assurance, 
the  government  greatly  steps  up  its  participation  to  assure 
the  final  end-product,  before  it  is  accepted. 

AFCMD  Quality  Assurance  policy  for  Software  ( AFCHDP 

74-3,  Apr  83)  clearly  dictates  this  orientation  as  inappro- 
priate  for  software. 

*  *  pi*  *•*  *  I!  ****************  * 

*  af be “"contract “award???  $0°n  33  posslble  * 

*  which  Jre  n«d  J°”puter  software  disciplines  * 

;  |oft„arerep^lud^.t0ThaeSrar?e:  T^sU l 

*  and°4 ““rluali tv°fl *  3>  Conf ^“ration  Management  * 

*  missino  ^  y  ^SVra"«.  If  one  of  these  is  * 

*  com???e^  —re®  13  tnotossod  likelihood  the  * 

*  comPuber  software  development  effort  will  * 

*  *Xfe*\fnce  Problems.  (AFCMDP  74-3).  * 

*****************  *  *  * 

AS  was  alluded  to  in  the  immediately  prior  section  and 

AFCMD  policy,  the  most  critical  activities  in  software  take 

place  in  the  design  and  testing  phases.  Not  only  is  this 

in  contrast  to  hardware  orientation,  but  also  the  production 

El-ase  for  software  is  so  trivial,  it  may  even  be  considered 
nonexistent . 
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In  place  of  a  significant  production  phase  for 
software,  there  is  instead  the  software  maintenance  phase. 
This  is  far  from  being  a  phase  where  corrections  to  software 
might  be  made  on  a  cost-effective  basis,  thus  taking  a 
marginally  functional  system  and  "tweaking  it  up"  at 
reasonable  cost.  Instead,  the  maintenance  phase  for 
software  has  been  repeatedly  shown  to  be  (Fairley,  1985): 

1)  the  most  costly  phase  of  the  software  life  cycle, 
and, 

2)  the  life-cycle  phase  in  which  any  given  error  costs 
the  most  to  correct. 

For  an  illustration  of  the  cost  of  correcting  errors  at 
different  points  in  the  Life-Cycle,  note  the  following 
figure: 


ERROR  CORRECTION 
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Phase  when  error 
is  detected 


1000  H 


100  — 


10— 


FIG.  TWO 

ECONOMY  OF  EARLY  ERROR 
CORRECTION 

(Martin.1985) 


maintenance 
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1.3.9  Comprehensive  Testing.  In  hardware  systems,  testing 
can  usually  be  nearly  exhaustive,  if  desired.  Certainly, 
there  are  trade-offs  and  not  all  testing  may  be  desireable 
relative  to  some  other  cost  factors,  but  testing  of 
components  and  complete  systems  is  generally  within  option. 
The  case  of  software  is  quite  different.  In  the  case  of 
computer  programs  that  are  large  enough  to  be  significant 
(and  the  programs  involved  with  weapons  systems  are  very 
large) ,  their  complexity  is  so  great  that  even  automated 
testing  by  intelligent  computer  programs  cannot  permit 
exhaustive  testing.  The  following  diagram  illustrates 
dramatically  how  the  complexity  of  a  conceptually  fairly 
simple  module  quickly  leads  to  impossible  requirements  for 
exhaustive  testing.  This  module  contains  in  excess  of  206 
trillion  unique  paths.  Assuming  a  mythical  test  generator 
that  could  devise  and  execute  tests  for  the  paths  in  such  a 
module  at  a  rate  of  one  per  millisecond,  it  would  take  over 
6,000  years  to  exhaustively  test  all  the  unique  paths. 
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FIG.  THREE 


OVER  206  TRILLION  UNIQUE  PATHS 
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As  a  result,  software  developers  are  left  with  the  task 
of  deciding  by  judgment  which  paths  of  a  program  are 
critically  important  enough  to  test.  This  tends  toward  the 
paradoxical,  since  this  judgment  is  necessary  because  the 
complexity  is  too  great  to  permit  analytical  understanding. 
Due  to  this  impossibility  of  exhaustive  testing  and  the  need 
to  rely  on  judgment-based  partial  testing,  software  systems 
have  a  quality  of  being  "haunted"  by  errors  which  are  almost 
certainly  there  (since  programming  is  highly  error  prone) , 
yet  which  have  not  been  detected  and  whose  criticality  and 
time  of  appearance  cannot  be  assessed.  Therefore,  the 
development  and  application  of  better  test  methods  must  be 
relentlessly  pursued. 
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2.0  LIFE-CYCLE  DIFFERENCES 

The  main  differences  between  software  and  hardware  in 
contract  administration  can  best  be  viewed  in  terms  of  the 
weapons  system  life  cycle  (WSLC)  and  the  software 
development  cycle  (SDC) .  The  overall  WSLC  consists  of  four 
sequential  and  essentially  discrete  phases: 

*  Concept  Exploration 

*  Demonstration  and  Validation 

*  Full-Scale  Development 

*  Production  and  Deployment 

The  first  two  phases  of  the  WSLC  are  more  study-like 
than  the  latter  two.  Concept  exploration  is  oriented  mostly 
toward  paper  study  and  computer  simulations,  examinations  of 
trade-off  effects,  feasibility,  etc.  Demonstration  and 
Validation  is  also  study-oriented,  but  involves  commitment 
of  resources  to  development  of  some  actual  subsystems  for 
study  and  testing.  The  outcome  of  the  Demonstration  and 
Validation  phase  is  the  capability  to  produce  a  fully 
functioning  and  reproducible  prototype  of  the  weapon  system. 

Full-scale  development  represents  the  first  complete 
development  of  enough  versions  of  the  weapons  system 
(including  training  and  support)  to  test  the  fully  operating 
system  and  move  into  development  of  large  volumes  of 
production  in  the  final  phase. 

Within  the  phases  of  the  WSLC,  hardware  and  software 
have  development  life  cycles  which  are  somewhat  comparable. 
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(NOTE:  THE  PHASES  ARE  SIMILAR  IN  FUNCTION  AND  HAVE  SIMILAR 

NAMES,  BUT  THEY  OPERATE  VERY  DIFFERENTLY  WITHIN  THE  WSLC. 

See  table,  following.) 
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SOFTWARE  DEVELOPMENT, 

CYCLE 

1.  System/Software 
Requirements  Analysis 

2.  Software  Requirements 
Analysis 

3.  Preliminary  Design 

4.  Detailed  Design 

5.  Coding,  Unit  Test  and 
Computer  Software 
Component  Integration 
Testing 

6.  Computer  Software 
Configuration  Item  Testing 


HARDWARE  DEVELOPMENT 

CYCUE 


1.  System/Hardware 
Requirements  Analysis 

2.  Hardware  Requirements 
Analysis 

3.  Preliminary  Design 

4.  Detailed  Design 

5.  Fabrication 


6.  Hardware  Configuration 
Item  Testing 


TABLE  TWO 
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The  terms  used  here  are  self-explanatory  and  the 
sequences  are  intuitively  meaningful,  as  well  as  the 
parallels  in  software  and  hardware  (some  kind  of  parallelism 
should  be  present,  since  they  develop  as  linked  subsystems 
of  a  parent  system) .  But  here  is  one  of  the  most  critical 
differences  for  contract  administration:  phases  of  the 
hardware  development  cycle  are  expected  to  proceed  in  a 
linear  fashion  with  virtually  no  overlap  —  by  contrast,  the 
phases  of  the  software  development  cycle  are  expected  a 
priori  to: 


Overlap  one  another  significantly 


Proceed  consequentially  with  an 
indeterminate  number  of  loops  between 
successive  phases  (this  implies  possible 
paths  from  phase  6 . 0  clear  back  to  phase 
1.0)  . 


Over span  the  phases  of  the  WSLC  (the 
hardware  cycle  is  expected  to  be 
finished  complete  within  a  given  phase 
of  the  WSLC) . 


(It  should  be  noted  that  both  the  hardware  development 
cycle  and  the  software  development  cycle  are  likely  to 
appear  in  somewhat  modified  forms,  as  is  appropriate,  in  the 
first  two  phases.) 

The  following  figure  attempts  to  illustrate  the 
complexity  of  these  relationships. 
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hardware  cycle 


software  cycle 


FIG.  FOUR 

COMPLEXITY  OF  SOFTWARE 


DEVELOPMENT  CYCLE 
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This  increased  complexity  of  the  software  development 
cycle  in  its  relationship  to  the  WSLC  has  probable 
implications  for  manpower  training  and  allocation  in 
contract  administration. 
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3.0  TRAINING  AND  SELECTION  OF  SOFTWARE  SPECIALISTS 

A  popular  myth  has  risen  in  the  defense  contracting 
industry  over  the  past  five  years.  The  need  for  software 
engineers  is  tremendous.  At  the  same  time,  avenues  of 
formal  education  and  experiential  resources  are  producing 
qualified  people  at  a  very  slow  rate.  Hence,  the  myth,  born 
of  need,  pressure  and  misconception:  "Just  take  a  bright 
engineer  who  has  had  a  few  programming  courses  (or  send  him 
to  a  few),  and  then  he  will  be  your  software  engineer." 
(Pressman,  1982.)* 

Generally,  it  doesn't  work  (exceptions  noted) .  First, 
because  software  engineering  and  computer  programming  are 
not  the  same  thing.  Second,  because  a  "few  courses"  in 
anything  will  not  change  the  hardware  engineering  mind-set. 
Third,  because  people  designated  as  "the  software  person" 
are  looked  to  for  leadership  and  authority.  It  is  a  very 
rare  individual  who  can  carry  off  that  role  with  inadequate 
training  and  experience. 

*  The  quotation  is  used  here  stylistically  and  does  not 

represent  a  direct  quote,  but  rather  a  paraphrase. 
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Regarding  government  contract  administration,  there 
seems  to  be  a  similar  myth.  Take  whoever  has  any  kind  of 
computer  training  on  his  transcript  (COBOL?,  DP  Systems? 

AI?)  and  designate  him  as  software  focal  point.  The  point 
here  is  not  to  criticize  software  focal  points,  or  to  dampen 
creativity  in  selection  practices  of  management,  or  to  deter 
anyone  from  seeking  and  accepting  career  challenge.  We  need 
to  examine  what  can  be  done  to  support  software  focal  points 
through : 

*  Development  of  their  leadership 
abilities  through  training 

*  Development  of  their  leadership  roles 

*  Technical  training  (for  them) 

*  Technical  training  (their  support  for 

their  colleagues) 

*  Other  areas 

Although  statements  have  been  made  here  about  casual 
and  general  assumptions  regarding  training  of  people  with 
special  roles  regarding  software,  it  is  clear  that  training 
and  personnel  development  needs  for  software  duties  are 
strong  throughout  the  contract  administration  organization. 
Hardly  anyone  is  expressing  competence  to  do  this  job.  The 
few  who  have  strong  competence  know  there  is  much  more  than 
they  can  do. 

(It  has  been  pointed  out  that  software  focal  points  are 
not  intended  to  be  software  specialists,  but  only 
administrative  functionaries.  However,  the  observation  in 
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AFPROs  is  that  software  focal  points  are  seen  by  themselves 
and  others  as  specialists  and  that  the  duties  of  this  role 
tend  to  expand  over  their  other  responsibilities.) 

One  of  the  assertions  made  frequently  when  AFPRO 
personnel  note  that  a  contractor  is  not  abiding  by  some 
commonly  accepted  (or  even  common  sense)  software  standard 
is  that  the  Program  Office  did  not  require  it  on  the 
contract;  hence,  it  was  not  enforceable. 

Assumedly,  with  the  advent  of  DOD-STD-2167,  much  of 
this  kind  of  situation  will  subside. .  Prior  to  DOD-STD- 
2167,  the  main  written  standards  for  software  were  MIL-S- 
52779A  and  MIL-STD-1679 .  These  were  inadequate  by 
themselves  and  so  many  of  the  specifics  necessary  to  bring  a 
Statement  of  Work  up  to  acceptable  criteria  had  to  be  added 
as  specific  items  to  each  contract.  Clearly,  this  left  much 
room  for  variance  and  ambiguity.  Perhaps  DOD-STD-2167, 
which  is  very  comprehensive,  articulated  and  up  to  date, 
will  improve  this  situation  to  a  degree.  Nevertheless,  it 
must  be  acknowledged  that  no  set  of  standards,  no  matter  how 
up  to  date  or  comprehensive,  can  substitute  for  a  well- 
written  SOW  or  the  oversight  of  development  by  a  competent 
government  staff. 

However,  there  is  some  reason  to  believe  that  part  of 
Program  Offices'  not  requiring  some  specifics  grew  out  of  a 
belief  that  the  AFPROs  lacked  sufficient  expertise  to 
support  and  enforce  the  requirements.  This  (along  with 
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other  direct  assertions)  suggests  less  than  effective  trust 
and  communication  between  Program  Office  and  AFPROs. 

One  of  the  needs  for  this  study  is  to  examine  ways  to 
insure  that  the  positive  potential  of  CAS  personnel  to 
improve  software  acquisition  is  not  dampened  by  historical 
lack  of  trust  and  communication.  New  patterns  of 
organizational  linkages  are  one  route  that  will  be  explored. 


Dennis  Blosser  &  Associates 
Final  Report 
Page  40 


4.0  NONDELIVERABLE  SOFTWARE 

Nondeliverable  software  refers  to  that  which  is 
intended  to  serve  as  a  support  in  developing  systems 
(Computer  Assisted  Design  software  or  Automatic  Test 
software,  for  example)  or  in  production  of  systems  (Computer 
Assisted  Manufacturing  software,  for  example) ,  but  is  not  a 
primary  end-product  (deliverable)  of  the  system  development 
process.  This  software  may  be  developed  by  a  contractor  as 
needed,  or  acquired  from  a  commercial  source  or  from  the 
government . 

Tremendous  advantages  are  available  through  the  use  of 
such  support  software.  In  the  case  of  test  software,  for 
example,  a  computer  program  can  often  generate  and  control 
the  execution  of  numbers  of  test  cases  which  would  represent 
a  number  of  man-years  effort  that  would  be  literally 
impossible  if  done  by  human  operators.  In  the  area  of 
production,  Computer  Assisted  Manufacturing  software  can 
control  processes  with  tolerances,  speeds  and  continuity 
that  could  not  be  approached  with  human  operators. 

However,  when  nondeliverable  software  is  used  in  the 
development  and  production  of  weapon  systems,  we  have  to  ask 
the  question: 

*  What  are  the  consequences  if 

nondeliverable  software  does  not 
function  properly? 

Obviously,  the  consequences  could  range  from  trivial  to  very 
serious  both  in  terms  of  cost  and  safety. 
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This  leads  to  the  conclusion  that,  even  though  some 
support  software  is  nondeliverable,  it  still  requires 
careful  management  and  surveillance  from  both  contractors 
and  the  government.  There  is  currently  very  little 
attention  being  paid  to  the  various  aspects  of  management  of 
nondeliverable  software.  Notably,  one  approach  to  the 
management  of  nondeliverable  software  did  surface  with  some 
consistency.  This  is  a  report  by  Major  George  Trevor  which 
surveys  the  concerns  of  nondeliverable  software  and  offers 
some  guidelines.  A  draft  version  of  this  report  is  included 
in  the  Appendix.  At  present,  there  is  a  level-of -awareness 
problem.  In  general,  the  lack  of  attention  to 
nondeliverable  software  comes  from  four  sources: 

(1)  The  very  heavy  and  specific  focus  on 
deliverable  software, 

(2)  Existence  of  clear-cut  standards  and 
procedures  for  deliverable  software  that  are 
lacking  for  nondeliverables, 

(3)  Assumption  that  if  it  is  nondeliverable,  its 
development  cannot  be  important  enough  to 
warrant  management  effort, 

(4)  Assumption  that  if  the  nondeliverable 
software  is  obtained  from  the  government  or  a 
commercial  source,  it  has  to  be  O.K. 

A  slightly  different  position  than  that  characterized  above 

was  also  observed.  It  may  be  characterized  as  "surveillance 

of  nondeliverable  software  is  important  and  MIL-STD-2167 

says  we  should  do  it,  but  we  don't  even  have  enough  manpower 

to  fulfill  the  needs  on  deliverable  software." 
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5.0  TECHNICAL  TASKS  IN  SOFTWARE  ACQUISITION 

5 . 1  Introduction  and  Background 

This  portion  of  the  research  was  undertaken  to 
investigate,  from  the  point  of  view  of  the  Air  Force 
Contract  Management  Division  (AFCMD) ,  the  following  general 
question:  How  can  AFCMD  and  the  System  Program  Offices 

(SPOs)  carry  out  their  joint  responsibilities  in  the 
acquisition  of  mission  critical  computer  resources  (MCCR) 
software  more  effectively? 

More  specifically,  under  this  general  question,  the 
following  questions  were  to  be  addressed: 


Should  policies,  practices,  etc.,  be  changed  for 
more  combined  effectiveness? 


What  skills/training  for  contract  administration 
personnel  would  support  recommended  changes? 


This  part  of  the  research  was  conducted  by  three  main 
methods  in  order  to  accumulate  basic  results  relevant  to  the 
research  questions: 

(a)  Review  of  Air  Force  documents  concerning  the 
formal  policies  for  operations  of  SPOs  and  AFCMD 
in  the  acquisition  of  MCCR  software, 

(b)  Interviews  with  SPO  and  AFCMD  personnel  at  five 
various  locations  each  throughout  the  country 
regarding  the  actualities  of  their  operations  (as 
compared  to  formal  policy) ,  and  their  viewpoints 
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and  suggestions  regarding  the  basic  questions  of 
this  research. 

(c)  Consideration  of:  (1)  discrepancies  between 

formal  policy  and  reported  actualities,  (2)  policy 
itself,  as  written,  and  (3)  expressed  attitudes 
and  suggestions  of  those  interviewed. 

The  existence  of  this  study  implies  that  there  is  a 
felt  need  to  improve  the  effectiveness  of  MCCR  software 
acquisition.  Studies  have  shown  (cf.  Booch,  1983)  that 
throughout  the  Department  of  Defense,  software  costs  are 
increasing  at  a  tremendous  rate  (six  billion  dollars  in 
1980,  predicted  to  exceed  32  billion  by  1990)  while  at  the 
same  time  software  is  frequently  behind  schedule  and  of  poor 
quality,  which  in  turn  impacts  overall  system  delivery 
schedule  and  increases  overall  system  costs. 

One  of  the  underlying  factors  in  this  state  of  affairs 
is  a  severe  shortage  of  personnel  trained  in  software 
engineering.  As  of  1984,  only  three  institutions  of  higher 
education  were  training  software  engineers  (Schaar,  1984) . 
Although  this  situation  is  improving,  it  is  predicted  that  a 
lag  and  shortage  of  well-trained  and  experienced  personnel 
will  be  a  problem  for  contractors  and  government  for  the 
foreseeable  future. 

Thus,  one  of  the  fundamental  sources  of  problems  in 
MCCR  software  acquisition  is  a  shortage  of  well-trained  and 
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experienced  personnel  that  strongly  affects  both  the 

contractors  and  the  government  acquisition  organizations. 

A  second  underlying  factor  in  the  problems  of  MCCR 
software  acquisition  is  time  pressure.  A  repeated 
observation  by  both  SPO  and  AFCMD  personnel  was  that  many 
critical  facets  of  the  MCCR  software  acquisition  process 
were  swept  over  by  the  time  pressure  of  schedules.  This 
ranged  through  the  whole  acquisition  process,  from  not 
having  time  to  prepare  Statements  of  Work  and  contractual 
requirements  with  sufficient  thoroughness,  to  not  being 
thorough  enough  with  pre-award  surveys,  to  glossing  over 
failures  in  quality  assurance,  to  glossing  over  software 
review  and  test  problems,  etc.,  all  because  of  time 
pressure. 

It  is  likely  that  the  shortage  of  well-trained  software 
personnel  interacts  with  the  time-pressure  problem  source. 

A  final  source  of  problems  should  be  considered  from 
the  AFCMD  point  of  view.  In  light  of  the  problematic  nature 
of  MCCR  software  acquisition,  the  shortage  of  well-trained 
personnel  in  this  area,  and  the  interactive  critical  effects 
of  time  pressure,  it  is  seriously  ineffective  and  a  waste  of 
government  funds  to  under  utilize  the  capacity  of  any 

component  of  the  government's  MCCR  software  acquisition 

organizations .  Nevertheless,  an  Inspector  General  study  in 
1981  showed  that  in  general,  SPOs  regarded  AFCMD  personnel 
as  significantly  lacking  capability  in  the  software  area. 
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AFCMD  headquarters  personnel  contend  that  this  is  currently 
not  the  case,  but  that  the  SPOs  still  regard  AFCMD  in  this 
fashion  and  thus  tend  to  under  utilize  their  capabilities. 

5.2  Identification  of  the  Tasks 

The  first  major  requirement  for  this  Phase  of  the  study 
was  to  identify  technical  tasks  to  be  performed  by  the 
government  when  acquiring  software  products.  This 
corresponds  to  task  4.3.1  of  the  Statement  of  Work. 

The  strategy  in  identifying  tasks  for  software 
acquisition  was  to  limit  the  scope  of  tasks  to  those  aspects 
of  the  software  acquisition  process  which  primarily  involve 
SPO  (System  Program  Offices)  and  CAS  (Contract 
Administration  Service)  activities.  This  strategy  was  used, 
since  the  context  of  the  State  of  Work  (SOW)  makes  it  clear 
that  these  aspects  of  software  acquisition  and  these  two 
agencies  are  the  focus  of  the  study. 

Tasks  in  software  acquisition  were  additionally 
selected  on  the  basis  of  these  further  criteria: 

(1)  Task  should  represent  an  appreciable  volume  of 
work  to  be  done,  usually  by  more  than  one 
individual . 

(2)  Task  should  represent  a  process  rather  than  an 
event . 

(3)  Task  should  be  software  dominated.  (This 
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distinction  is  difficult,  not  always  clear,  and 
probably  arguable) . 

(4)  Task  should,  as  much  as  possible,  represent  a 

description  of  "what  is  to  be  done"  rather  than 
"how  to"  or  criteria  or  policy.  (Once  again,  a 
sometimes  less-than-clear  distinction.) 

Many  sources  were  reviewed  and  influence  the  selection 
of  tasks  specified  for  this  study  (see  References) .  In  the 
final  analysis,  it  was  decided  to  move  as  close  as  possible 
to  concrete  tasks  and  away  from  generic  or  more  abstractly 
stated  tasks.  For  this  reason,  and  due  to  ultimately 
arbitrary  choices,  the  documents  which  were  relied  on  to  the 
greatest  extent  in  the  final  analysis  were: 

(1)  Airborne  Systems  Software  Acquisition  Engineering 

Guidebook  for  Contracting  for  Software 

Acquisition.  ASD/ENAI. 

(2)  Contracting  for  Software  Acquisition.  Software 
Acquisition  Guidebook  Series,  ASD/ENE 

(3)  Contract  Administration  of  Mission  Critical 
Computer  Resources  in  Systems  Acquisition.  Joint 
Contract  Administration  Coordinating  Council 
Guidebook,  draft. 

(4)  Captain  Dennis  W.  Smith,  personal  communication. 

(5)  Guide  to  the  Management  of  Embedded  Computer 

Resources .  Space  Division. 
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Table  Three,  below,  contains  the  89  tasks  identified 
for  software  acquisition.  The  context  here  is  software,  so 
that  is  specifically  implied  for  each  task,  even  if  not 
stated.  The  order  of  the  tasks  shown  is  in  roughly 
chronological  order.  However,  strict  chronological  order 
and  sequentiality  must  not  be  expected. 

While  Table  Three  is  intended  to  be  comprehensive, 
there  are  some  restrictions  which  must  be  made  in  this 
sense.  First,  there  is  the  problem  of  diversity.  Although 
one  criterion  in  selecting  tasks  for  this  analysis  was 
concreteness,  this  criterion  itself  (though  useful) 
mitigates  against  comprehensiveness.  Software  applications 
in  embedded  systems  are  so  diverse  that  any  given  project 
may  have  a  great  deal  of  uniqueness  in  its  software.  For 
this  reason,  the  only  way  to  cover  all  possibilities  is  to 
present  tasks  that  are  extremely  generic  or  abstract. 
Hopefully,  a  useful  middle-ground  has  been  achieved  here. 
However,  any  given  application  may  certainly  require  tasks 
not  represented  here. 

The  second  problem  for  comprehensiveness  involves  the 
problems  of  the  extent  of  the  effects  of  software.  With 
increasing  applications  of  computers  and  complexity  of 
software,  it  is  difficult  to  delimit  the  extent  of  the 
effects  of  software.  The  selection  of  tasks  for  this  study 
implies  a  delimitation  of  the  systemic  influences  of 
software.  It  is  noted  that  this  is  sensible  but  arbitrary. 
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TABLE  THREE 

TECHNICAL  TASKS  IN  THE  ACQUISITION 
OF  SOFTWARE 


1.  Use  the  following  checklist  as  appropriate  to 
prepare  RFP  package. 

Checklist 

A  -  Introduction 

(1)  Define  software  to  be  developed 

(2)  State  function  of  software 

(3)  Identify  users 

B  -  Scope 

(1)  Define  the  development  phase 

(2)  Define  technical  objectives  in  each  phase 

(3)  Brief  previous  history 

(4)  Reference  documents 
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TABLE  THREE  CONTINUED 
TECHNICAL  TASKS  IN  THE  ACQUISITION 
OF  SOFTWARE 


Checklist  Continued 

C  -  Tasks 

(1)  Specific  requirements  for  contractor 

(2)  No  "to  be  determined"  or  informal  tasking 

(3)  "Identify"  and  "design"  tasks  giving 

specific  module  functions 

(4)  Requirements  for  design  management 

(5)  Requirements  for  test  methodology  planning, 

execution  and  evaluation 

(6)  Requirements  (if  any)  for  independent 

verification  and  validation 

(7)  Requirements  for  definition  of  acceptance 

certification 

(8)  How  user  training  will  be  supported 

(9)  Requirements  for  operational  simulations 

and/or  rehearsals 

(10)  Requirements  for  methodology  and  capacity 
to  modify  and  upgrade  software 
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TABLE  THREE  CONTINUED 
TECHNICAL  TASKS  IN  THE  ACQUISITION 
OF  SOFTWARE 


Checklist  Continued 

D  -  Review  compliance  documents  for  applicability 

•  Work  Breakdown  Structure  (MIL-STD-881A) 

•  Configuration  Management  (MIL-STD-483) 
(tailored  as  required) 

•  Specification  Practices  (MIL-STD-490) 

(tailored  as  required) 

•  Technical  Reviews  and  Audits  for  Systems, 

Equipment  and  Computer  Programs  (MIL- 
STD-1521A)  (tailored  as  required) 

•  Software  Quality  Assurance  Program 
Requirements 

(MIL-S-52779A) 

•  Specifications,  Types  and  Forms  (refers  to 

MIL-STD-483/490)  (MIL-S-83490) 

•  Software  Development  Standard  (MIL-STD- 

2167A)  (tailored  as  required) 

-  End  of  Checklist  - 


Dennis  Blosser  &  Associates 
Final  Report 
Page  51 


TABLE  THREE  CONTINUED 
TECHNICAL  TASKS  IN  THE  ACQUISITION 
OF  SOFTWARE 


2 .  Request  inclusion  of  all  detailed  assumptions  and 

rationale  for  submitted  software  cost  estimate. 

3 .  Request  that  the  proposal  includes  appropriate  reuse 

of  appropriate  software  previously  developed  by 
the  contractor. 

4.  Refine  the  statement  of  work  by  internal  review  and 

revision. 

5.  ALTERNATELY  (for  large  projects  >  $1M) 

*  Compile  a  list  of  interested  bidders. 

*  Circulate  draft  RFP  and  get  questions  from 

bidders. 

*  Hold  a  briefing  for  interested  bidders  to  answer 

(anonymous)  questions. 
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TABLE  THREE  CONTINUED 
TECHNICAL  TASKS  IN  THE  ACQUISITION 
OF  SOFTWARE 


6.  Assess  whether  to  separate  conceptual  validation  and 

development  phases  into  separate  contracts. 

7.  Understand  the  contractual  data  items  relevant  to 

the  specific  procurement. 

8.  Customize  the  application  of  relevant  data  items  to 

the  specific  environment  and  acquisition 
strategy. 

9.  Conduct  a  pre-award  survey.  Review  a  guidebook  such 

as  "Pre-Award  Survey  for  Contractor  Software 
Developments"  (SD/ALR  Guide,  June  1982) . 

10.  Submit  the  pre-award  survey  document  to  bidders  upon 

receipt  of  their  proposals  and  allow  a  minimum 
of  four  weeks  for  them  to  organize  their 
responses  prior  to  conducting  the  survey. 
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TABLE  THREE  CONTINUED 
TECHNICAL  TASKS  IN  THE  ACQUISITION 
OF  SOFTWARE 


11.  Prepare  a  comprehensive  set  of  questions  for  each 

major  software  area  relevant  to  the  contract  and 
tailored  to  the  contractor’s  environment. 

12 .  Conduct  the  interviews  on  site  with  allowance  for 

the  answers  to  questions  to  spontaneously 
generate  new  questions  or  areas  of 
investigation/ audit  as  appropriate  (i.e.,  highly 
skilled  and  software-experienced  interviewers) . 

13.  Assemble  a  team  of  highly  skilled  software 

management  experts  to  conduct  the  pre-award 
survey . 

14.  Assess  the  survey  results  and  submit  the  assessment 

to  the  appropriate  AFPRO  or  DCAS . 
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TABLE  'THREE  CONTINUED 


TECHNICAL  TASKS  IN  THE  ACQUISITION 


OF  SOFTWARE 


15.  AFPRO  or  DCAS  has  responsibility  to  finalize  pre¬ 

award  report  and  provide  the  report  to  the  SPO. 

16.  AFPRO/DCAS  can  conduct  software  pre-award,  but  SPO 


has  final  responsibility  for  assembly  of  total 
team. 


17.  If  the  contractor's  (or  contractor's  division) 


average  net  profit  is  small  relative  to  the  size 
of  the  software  contract  and  the  contract  is 
"firm,  fixed-price,"  the  financial  capability 
aspect  of  the  pre-award  survey  must  be 
critically  considered  along  with  the  software 
component.  This  is  because  estimates  of 
software  costs  are  extremely  volatile,  and 
actual  costs  may  exceed  estimates  as  a  factor  of 
ten.  This  situation  poses  a  possibly  severe 
risk  for  receiving  the  desired  product. 
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TABLE  THREE  CONTINUED 
TECHNICAL  TASKS  IN  THE  ACQUISITION 
OF  SOFTWARE 


18.  Obtain  general  assistance  and  information  on 

contractor's  general  capabilities  in  software 
areas  from  AFCMD.  Augment  AFCMD  capability  with 
SPO  personnel  during  conduct  of  the  pre-award 
survey . 

19.  Establish  channel  of  communication  between  SPO  and 

AFCMD  specific  to  new  program/project. 

20.  Determine  requirements  necessary  for  letters  of 

delegation  to  subcontractors. 

21.  Conference  contract  administration,  quality 

assurance,  engineering,  and  other  AFCMD  support  as 
necessary  to  ensure  flow  of  information  to  SPO. 

22.  Identify  contract  requirements  for  performance 
measurements . 
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TABLE  THREE  CONTINUED 
TECHNICAL  TASKS  IN  THE  ACQUISITION 
OF  SOFTWARE 


23.  Define  the  relationships  and  responsibilities  of  the 

SPO  and  AFPRO  for  the  contract,  including  a 
memorandum  of  agreement,  if  appropriate. 

24.  General  review  of  draft  RFP  for  consideration  of  all 

components  to  ensure  enforceability  of  contract 
requirements,  proper  subcontract  or  management, 
complete  pricing  requirements,  and  consideration 
of  delegation  of  some  monitoring  efforts  to  AFPRO 
by  SPO  not  already  included  in  the  RFP. 

25.  Review  of  the  draft  SOW  to  determine  whether  the 

management  requirements,  software  development  of 
criteria,  procedures,  referenced  to  CDRLs  and 
specifications,  and  other  consideration  provide 
adequate  means  to  properly  monitor  the  contract. 

26.  Review  of  the  draft  CDRLs  to  determine  the 

effectiveness  of  the  DIDs  and  detailing  of  CDRLs 
provide  adequate  means  to  properly  monitor  the 
contract . 
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TABLE  THREE  CONTINUED 
TECHNICAL  TASKS  IN  THE  ACQUISITION 
OF  SOFTWARE 


27.  Review  of  the  draft  RFP  to  determine  the 

reasonableness  of  the  overall  time  duration; 
whether  the  delivery  schedule  for  various  plans 
supports  their  valid  useability;  identification  of 
critical  needs  for  long  lead  times; 
appropriateness  of  milestones  and  sequences;  areas 
of  high  potential  risk  to  schedule  maintenance. 

28.  Review  of  the  draft  RFP  to  determine  the  presence  of 

appropriate  and  current  versions  of  standards  and 
specifications  and  to  check  tailoring  against 
requirements  for  valid  administration  of  contract. 

29.  Prepare  and  review  sample  contract  to  ensure 

inclusion  of  necessary  clauses  and  forms,  deletion 
of  unnecessary  forms  and  clauses,  proper  cross 
references  to  MIL-specs  included  in  SOW/CDRL,  and 
other  technical  and  clerical  details. 
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TABLE  THREE  CONTINUED 
TECHNICAL  TASKS  IN  THE  ACQUISITION 
OF  SOFTWARE 


30.  Prepare  source  selection  plan. 

31.  Obtain  approval  of  source  selection  plan  from  Source 

Selection  Authority. 

32.  Establish  criteria  for  evaluation. 

33.  Establish  factors  and  standards. 

34.  Prepare  an  independent  cost  estimate. 

35.  Evaluate  proposals. 

36.  Determine  the  competitive  range. 

37.  Identify  deficiencies  in  proposals. 
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TABLE  THREE  CONTINUED 
TECHNICAL  TASKS  IN  THE  ACQUISITION 
OF  SOFTWARE 


38.  Carry  out  written/oral  discussions  with  bidders. 

39.  Obtain  final  and  definite  contracts. 

40.  Negotiate  prices  with  bidders. 

41.  Complete  evaluation. 

42.  Prepare  Source  Selection  Evaluation  Board  Report. 

43.  Prepare  Source  Selection  Advisory  Council  Report. 

44.  Render  final  Source  Selection  Authority  decision. 

45.  Document  the  source  selection  decision. 
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TABLE  THREE  CONTINUED 
TECHNICAL  TASKS  IN  THE  ACQUISITION 
OF  SOFTWARE 


46.  Evaluation  of  proposal  by  CAS  representatives  based 

on  past  experiences  with  administration  of 
contracts  with  similar  requirements  and 
characteristics . 

47.  Participation  in  Source  Selection  panels  by  CAS 

representatives  specifically  in  regard  to 
historical  knowledge  of  abilities  of  bidders  under 
consideration  and  experience  with  technical  and 
management  aspects  of  bidders'  proposals. 

48.  Review  contract  as  awarded  to  determine  presence  of 

software  functional  areas  such  as  Work  Breakdown 
Structure,  Military  Standards  and  Specifications, 
Contract  Data  Requirements  Lists,  Software 
Development  Plan,  Software  Configuration 
Management  Plan,  Software  Quality  Assurance 

49 .  Identify  and  document  potential  contract 

administration  problems  due  to  terms  and 
conditions  of  the  contract. 


Plan. 
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TABLE  THREE  CONTINUED 
TECHNICAL  TASKS  IN  THE  ACQUISITION 
OF  SOFTWARE 


50.  Insure  that  formal  agreements  (such  as  Memorandum  of 

Agreement)  exist  between  SPO  and  CAS  identifying 
their  specific  duties  and  responsibilities 
concerning  software. 

51.  Develop  an  internal  CAS  plan  for  assignment  of 

manpower,  test  and  review  participation, 
inspection  points,  and  communication 
responsibilities . 

52.  Identify  and  develop  plan  for  desired  contract 

administration  activities  for  subcontracted 
software. 

53.  Optional  SPO  and  CAS  Post-award  Orientation 

Conference  of  software  concerns  and 
responsibilities . 

54.  Ongoing  evaluation  and  review  of  contractor's 

management  system  for  software  development. 
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TABLE  THREE  CONTINUED 


TECHNICAL  TASKS  IN  THE  ACQUISITION 


OF  SOFTWARE 


55.  Monitor  critical  subcontracts  for  flow-down  of 

software  contract  requirements. 

56.  Monitor  cost,  schedule,  and  performance  of  software 

development  tasks. 

57.  Resolve  technical  and  management  problems  in 

software  development  process. 

58.  Review  and  evaluate  software  Engineering  Change 

Proposals. 

59.  Review  contract  modifications  for  any  possible 

effects  on  monitoring  software  development. 

60.  Monitor  development  and  deliveries  of  software 

Contract  Data  Requirements  List  documents. 
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TABLE  THREE  CONTINUED 
TECHNICAL  TASKS  IN  THE  ACQUISITION 
OF  SOFTWARE 


61.  Monitor  and  review  compliance  with  automated  test 

system  standards  and  documentation. 

62.  Audit  contractor  compliance  with  software 

requirements  for  configuration  management  and 
quality  assurance. 

63.  Review  and  report  to  SPO  on  contractor's  technical 

planning  and  management  of  software  development 
prior  to  each  formal  review. 

64.  Follow  up  all  action  items  designated  by  SPO  as  a 
result  of  each  formal  review. 

Evaluate  the  adequacy  of  the  contractor's  system 
requirements  definition  through  the  formal  System 
Requirements  Review. 


65. 
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TABLE  THREE  CONTINUED 
TECHNICAL  TASKS  IN  THE  ACQUISITION 
OF  SOFTWARE 


66.  Evaluate  the  completeness  of  the  allocated  technical 

requirements  and  the  technical  and  management 
risks  of  the  software  development  through  the 
formal  Systems  Design  Review. 

67.  Assess  the  contractor's  system  development  tools  and 

management  and  technical  abilities  to  carry  out 
the  necessary  development  functions. 

68.  Review  the  Computer  Software  Configuration  Item's 

functional,  performance,  data  base,  qualification, 
interface,  and  delivery  requirements  through  the 
formal  Software  Specification  Review. 

69.  Assess  proceeding  with  contractor's  designs  for  each 

Computer  Software  Configuration  Item  through  the 
formal  Preliminary  Design  Review. 
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TABLE  THREE  CONTINUED 
TECHNICAL  TASKS  IN  THE  ACQUISITION 
OF  SOFTWARE 


70.  Determine  preparedness  to  proceed  to  the  coding 

phase  through  evaluation  of  the  contractor ' s 
Software  Detailed  Design  through  the  formal 
Critical  Design  Review. 

71.  Monitor  software  coding  phase. 

72.  Monitor  software  integration  and  test. 

73.  Evaluate  test  procedures,  tools,  facilities, 
personnel,  configuration  control  procedures,  and 
other  necessary  factors  to  determine  preparedness 
for  qualification  testing  through  the  formal  Test 
Readiness  Review. 

Evaluate  the  performance  of  the  software,  the 
adherence  to  approved  test  procedures,  and  the 
accuracy  of  test  results  through  Formal  Testing. 


74. 
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TABLE  THREE  CONTINUED 
TECHNICAL  TASKS  IN  THE  ACQUISITION 
OF  SOFTWARE 


75.  Evaluate  the  validity,  compliance,  and  accuracy  of 

test  results  of  the  software  integrated  into  the 
system  through  formal  System  Integration  and  Test. 

76.  Document  the  completeness  of  meeting  requirements 

and  specifications,  and  testing  of  each  Computer 
Software  Configuration  Item  and  Software  Product 
through  formal  Functional  and  Physical 
Configuration  Audits. 

77.  Monitor  planning  and  execution  of  Independent 

Verification  and  Validation. 

78.  Monitor  development  of  automated  tools  used  to 

prepare  technical  manuals,  operating  procedures, 
engineering  drawings,  etc.,  to  assure 
compatibility  with  end-user  systems. 
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TABLE  THREE  CONTINUED 
TECHNICAL  TASKS  IN  THE  ACQUISITION 
OF  SOFTWARE 


79.  Monitor  assurance  that  configuration  management  and 

historical  data  systems  are  maintained. 

80.  Determine  compatibility  of  maintenance  procedures  to 

human  factors  and  equipment. 

81.  Evaluate  training  requirements  for  operation  and 

support  of  the  software  system. 

82.  Evaluate  software  system  documentation  to  assure 

support  of  post-acceptance  maintenance. 

83.  Evaluate  logistical  support  to  system  users  for 

post-acceptance  maintenance. 

84.  Complete  software  acceptance  procedures. 
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TABLE  THREE  CONTINUED 
TECHNICAL  TASKS  IN  THE  ACQUISITION 
OF  SOFTWARE 


85.  Review  the  Configuration  Management  Plan  for  post¬ 

development  software  support. 

86.  Review  the  Software  Quality  Assurance  Plan  for  post¬ 

development  software  support. 

87.  Review  Software  Test  Plans  for  post-development 

software  support. 

88.  Determine  that  proposed  engineering  changes, 
deficiencies,  and  latest  defects  in  software  are 
corrected  by  the  contractor  and  the  Configuration 
Management  Plan  adjusted  accordingly. 

Establish  a  Configuration  Management  Plan  for  post¬ 
development  software  support. 


89. 
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The  tasks  in  this  table  are  drawn  from  sources  which 
predate  MIL-STD-2167 .  However,  they  are  generic  enough  that 
it  is  unlikely  that  2167  would  have  much  effect  on  them. 
Also,  the  policy  which  these  tasks  reflect  will  probably  not 
be  reconsidered  in  the  light  of  2167  for  some  time,  since 
2167  is  a  contractor  standard. 
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6.0  RELATION  OF  TASKS  TO  SPO  AND  CAS 

The  second  major  requirement  of  this  phase  of  the  study 
was  to  identify  those  technical  tasks  more  appropriately 
performed  by  SPO  personnel  and  those  more  appropriately 
performed  by  the  in-plant  CAS  personnel.  This  corresponds 
to  task  4.3.2  in  the  SOW  for  the  study . 

It  creates  problems  to  devise  exclusive  divisions  of 
tasks,  such  as  those  in  Table  Three,  between  agencies  such 
as  SPO  and  CAS,  i.e.,  the  acquisition  and  development  of 
software  is  complex  and  nonlinear.  It  is  not  reasonable  to 
expect  that  a  totally  simple,  clear-cut  division  of  tasks 
would  be  feasible.  The  notion  of  concise  separation  is 
certainly  appealing  administratively.  But  in  terms  of 
effectively  acquiring  the  development  of  complex  software 
systems,  this  would  amount  to  a  useless  oversimplification. 

In  order  to  move  toward  a  useful  separation  and 
coordination  of  tasks  which  would  result  in  effective 
software  acquisition,  the  contractor  has  derived  a 
simplified  schema  for  the  relationships  among  tasks  and  the 
two  agencies  involved  in  the  acquisition  of  software.  The 
aim  here  is  to: 

(1)  produce  clarification, 

(2)  support  analysis  and  evaluation, 

(3)  establish  effective  roles, 
acknowledge  interrelated  roles,  and 


(4) 
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(5)  avoid  oversimplification. 

For  this  purpose,  a  very  simple  schema  was  established 
for  the  assignment  of  task  relationships  to  SPO  and  CAS. 

The  schema  is  shown  in  Table  Four. 

The  schema  was  kept  as  simple  as  possible,  which  seems 
appropriate  for  the  present  study.  However,  if  these 
recommendations  are  considered  for  implementation,  it  may  be 
that  a  more  articulated  schema  would  be  useful. 
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TABLE  FOUR 

SCHEMA  OF  TASK-RELATIONSHIPS 
FOR  SPO  AND  CAS 


Code  Use  In  Table  III 


Meaning 


I  Primary  Responsibility 


II  Must  be  Consulted 


III  May  be  Consulted 


No  significant  responsibility 
(Does  not  imply  exclusion  or 
that  responsibility  may  not  be 
enlarged  by  agreement.) 
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Table  Five  shows  the  tasks  previously  identified  for 
software  acquisition  displayed  against  task-relationship 
codes  for  SPO  and  CAS.  In  Table  Five,  only  a  key  word  or 
phrase  is  given  for  each  task.  However,  the  number  of  each 
task  in  Table  Five  corresponds  to  the  number  of  the  full 
task  description  in  Table  Five,  for  reference. 

There  is  a  great  deal  of  information  contained  in 
Table  Five,  and  of  course,  the  distinctions  it  makes  should 
be  considered  in  detail  by  anyone  undertaking  to  implement 
its  recommendations  as  policy.  However,  without  addressing 
each  item  individually,  the  general  implications  of  Table 
Five  can  be  usefully  summarized. 

Table  Five  reflects  an  increased  amount  of  required 
consultation  of  the  CAS  by  the  SPO  than  is  currently 
reflected  in  written  policy  and/or  practice.  This  is 
especially  true  in  the  areas  of  pre-award  survey  and  RFP 
development.  Interviews  conducted  in  the  study  indicate 
that  under  current  practice,  valuable  knowledge  developed  by 
CAS  agencies  through  their  long-term  relationships  with  day- 
to-day  contractor  activities  is  under-  utilized  and  wasted  in 
these  areas.  If  followed,  the  recommendations  of  Table  Five 
should  alleviate  this  situation,  which  should  not  be  allowed 
to  continued  in  the  critical,  costly,  and  problem-ridden 
area  of  software  acquisition. 
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TABLE  FIVE 

TASK-RELATIONSHIPS  CODED  FOR 
SPO  AND  CAS 


Task 


SPO  CAS 


1.  RFP  Package  Checklist 


I 


2.  Reqaest  cost  assumptions 


I 


3 .  Previously  developed 
software  reuse 


I 


4.  Internal  SOW  refinement  I 


5.  Alternate  bidder  input 


I 
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TABLE  FIVE  CONTINUED 
TASK-RELATIONSHIPS  CODED  FOR 
SPO  AND  CAS 

Task  SPO  CAS 

6.  Assess  contract  separation  I  III 

7.  Understand  data  items  I  - 

8.  Customize  data  items  I  III 

9.  Conduct  pre-award  survey  I  II 

10.  Pre-award  survey  to  bidders  I  - 

11.  Prepare  pre-award  questions  I  III 
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TABLE  FIVE  CONTINUED 
TASK-RELATIONSHIPS  CODED  FOR 
SPO  AND  CAS 

Task  SPO  CAS 

12.  On-site  interviews  I  II 

13 .  Assemble  software  expert  team  I  II 

14.  Submit  assessment  to  CAS  I  — 

15.  Finalize  pre-award  report  -  I 

16.  CAS  pre-award  survey  I  II 
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TABLE  FIVE  CONTINUED 
TASK-RELATIONSHIPS  CODED  FOR 
SPO  AND  CAS 


Task 


SPO  CAS 


17.  Evaluate  contractor 
profit-volume/risk 


I 


18.  Input  from  AFCMD  on 
pre-award 


II  I 


19.  Establish  SPO/AFCMD 
communication 


I 


I 


20.  Subcontractor  letter 
requirements 


I 


II 
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TABLE  FIVE  CONTINUED 
TASK-RELATIONSHIPS  CODED  FOR 
SPO  AND  CAS 


Task  SPO  CAS 

21.  AFCMD  internal  conference  -  I 

22.  Contract  requirements  for 

measurement  III  I 

23.  SPO/CAS  relationships  I  I 

24.  Review  draft  RFP  III  I, II 


25.  Review  draft  SOW 


III 


I, II 
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TABLE  FIVE  CONTINUED 
TASK-RELATIONSHIPS  CODED  FOR 
SPO  AND  CAS 


Task  SPO  CAS 

26.  Review  CDRLs  III  I ,  II 

27.  Review  draft  RFP 

scheduling  III  I, II 

28.  Review  standards, 

specifications,  tailoring  III  I ,11 

29.  Review  sample  contract 

validity  III  I /II 


30.  Source  selection  plan 


I 


III 
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TABLE  FIVE  CONTINUED 
TASK-RELATIONSHIPS  CODED  FOR 
SPO  AND  CAS 


Task  SPO  CAS 

31.  Source  selection  plan 

approval  I  - 

32.  Establish  evaluation 

criteria  I  III 

33.  Establish  factors  and 

standards  I  III 

34.  Independent  cost  estimate  I  II 

35.  Evaluate  proposals  I  II 
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TABLE  FIVE  CONTINUED 
TASK-RELATIONSHIPS  CODED  FOR 
SPO  AND  CAS 


Task  SPO  CAS 

36.  Determine  competitive 

range  I  111 

37.  Identify  deficiencies  I  II 

38.  Confer  with  bidders  I  IH 

39.  Final  contracts  I 

40.  Negotiate  prices  I  111 
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TABLE  FIVE  CONTINUED 
TASK-RELATIONSHIPS  CODED  FOR 
SPO  AND  CAS 


Tasjc 


CAS 


41.  Complete  evaluation 


I  III 


42.  Source  Selection  Evaluation 
Board  report 


I 


43.  Source  Selection  Advisory 
Council  report 


I 


44.  Final  Decision 


I 


45.  Document  Decision 


I 
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TABLE  FIVE  CONTINUED 
TASK-RELATIONSHIPS  CODED  FOR 
SPO  AND  CAS 


Task  SPO  CAS 

46.  CAS  proposal  evaluation  -  I,  II 

47.  CAS  in  Source  Selection 

Panels  I  III 

48 .  Review  awarded  contract  -  I 

49.  Identify  potential 

contract  problems  -  I 

50.  Insure  specificity  of 

SPO/ CAS  agreements  I  I 
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TABLE  FIVE  CONTINUED 
TASK-RELATIONSHIPS  CODED  FOR 
SPO  AND  CAS 

Task  SPO  CAS 

51.  Internal  CAS  plan  III  I 

52 .  CAS  subcontract  plan  III  I 

53.  Post-award  orientation 

conference  II  I 

54 .  Ongoing  management 

evaluation  II  I 

55.  Subcontract  flowdown  II  I 
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TABLE  FIVE  CONTINUED 
TASK-RELATIONSHIPS  CODED  FOR 
SPO  AND  CAS 


Task  SPO  CAS 

56.  Monitor  software 

development  measures  II  I 

57.  Technical  and  management 

problem  resolution  I, II  I /II 

58 .  Evaluate  engineering 

change  proposals  I ,11  I ,11 

59.  Review  contract  modifications  III  I 


60.  Monitor  CDRL  delivery 


III 


I 
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TABLE  FIVE  CONTINUED 
TASK-RELATIONSHIPS  CODED  FOR 
SPO  AND  CAS 

Task  SPO  CAS 

61.  ATS  standards 

compliance  III  I 

62.  CM  and  QA  compliance 

audits  III  I 

63 .  Pre-review  reports  II  I 

64.  Follow-up  review 

action  items  II  I 
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TABLE  FIVE  CONTINUED 
TASK-RELATIONSHIPS  CODED  FOR 
SPO  AND  CAS 

Task.  SPO  CAS 

65 .  systems  Requirements 

Review  I  III 

66.  System  Design  Review  I  II 

67 .  Assess  development 

capabilities  II  I 

68.  Software  Specification 

Review  I  II 

69 .  Preliminary  Design  Review  I  II 
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TABLE  FIVE  CONTINUED 
TASK-RELATIONSHIPS  CODED  FOR 
SPO  AND  CAS 

Task  SPO  CAS 

70.  Critical  Design  Review  I  II 

71.  Coding  monitor  II  I 

72.  Integration  and 

test  monitor  II  I 

73.  Test  Readiness  Review  I  II 

74.  Formal  Testing  I  II 
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TABLE  FIVE  CONTINUED 
TASK-RELATIONSHIPS  CODED  FOR 
SPO  AND  CAS 

Task  SPO  CAS 

75.  System  Integration  Test  I  II 

76.  Monitor  Configuration  Audits  II  I 

77.  Monitor  IV  &  V  II  I 

78.  Monitor  tools 

compatibility  II  I 

79.  Assure  CM  maintenance  II  I 
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TABLE  FIVE  CONTINUED 
TASK-RELATIONSHIPS  CODED  FOR 
SPO  AND  CAS 

Task  SPO  CAS 

80.  Evaluate  Human  and 

equipment  compatibility  II  I 

81.  Evaluate  Training 

Requirements  II  I 

82.  Evaluate  documentation  II  I 

83.  Evaluate  logistical 

support  II  I 

84.  Software  acceptance  II  I 
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TABLE  FIVE  CONTINUED 
TASK-RELATIONSHIPS  CODED  FOR 
SPO  AND  CAS 


Task  SPO  CAS 


85.  Review  CM  for 

post-development  II  I 

86.  Review  SQA  plan  for 

post-development  II  I 

87.  Review  Test  Plans 

post-development  II  I 

88.  Determine  corrections 

and  updates  II  I 
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TABLE  FIVE  CONTINUED 
TASK-RELATIONSHIPS  CODED  FOR 
SPO  AND  CAS 


Task 


SPO  CAS 


89.  Review  CM  plan  for 
post-development 


II 


I 
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Another  general  effect  of  Table  Five  is  to  recommend 
more  compulsory  participation  of  CAS  personnel  in  formal 
reviews  during  the  development  process.  As  specified  here, 
required  CAS  participation  would  begin  with  the  System 
Design  Review  (Table  Five,  number  66) .  Such  early 
involvement  is  necessary  in  order  for  CAS  personnel  to  do 
their  jobs  effectively  through  the  later  stages  of  the 
software  development  process.  Also,  CAS  familiarity  with 
day-to-day  problems  of  software  development  offers  the 
potential  to  predict  and  prevent  potential  problems  earlier 
in  the  development  process.  This  offers  the  possibility  of 
great  dollar  and  time/schedule  savings  to  the  government. 

A  final  phrase  in  the  SOW  for  this  component  of  the 
study  is  "for  each  task  assignment,  give  rationale  and 
identify  benefits  accrued  to  the  government."  To  do  this 
for  each  task  assignment  would  be  extremely  redundant,  as 
the  rationale  for  each  assignment  (in  that  it  differs  from 
current  policy/practice)  is  pretty  much  the  same  and  is 
outlined  in  the  immediately  preceding  paragraphs  along  with 
potential  benefits.  In  short,  current  policy  and  practice 
fails  to  utilize  available  CAS  potential  to  produce  better 
software  contracts  and  fails  to  involve  CAS  personnel  early 
enough  to  support  their  potential  contributions  to  early 
problem  prevention  and  correction.  Since  software 
development  is  so  costly  in  dollars  and  schedule  and  so 
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burdened  with  errors  and  setbacks,  fuller  use  should  be  made 
of  these  obvious  potentials. 

Probably  the  most  specific  and  clear-cut  policy 
statement  regarding  the  task-relationships  in  Table  Five 
appears  in  AFSCR  800-42: 

"3.C.  System  Program  Office  will  —  ... 

(8)  Involve  AFCMD  (before  initial  contract  award) 
or  the  AFPRO  in  the  development  and  review  of  the 
following  program  documents,  and  as  a  participant  in 
the  following  precontract  activities.  Except  for  sole 
source  acquisitions  not  administered  by  AFCMD,  AFCMD 
should  be  involved  to  help  make  sure  the  contract  can 
be  properly  administered,  even  if  the  resulting 
contracts  may  be  administered  by  another  CAO  agency. 

(a)  Program  management  plans  (PMP) . 

(b)  Acquisition  plans  (AP) . 

(c)  Statements  of  work  (SOW) . 

(d)  Requests  for  proposals  (RFP) ,  including 
draft  RFPs. 


(e)  Drafts  of  proposed  contracts. 

(f)  Subcontract  management  plans  (SMP) . 

(g)  Proposal  evaluations. 

(h)  Source  selections. 

(i)  Contract  negotiations. 

(j)  Business  strategy  panels  (BSP). 

(k)  Solicitation  review  panels  (SRP) . 
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(l)  Manufacturing  management  and  production 
capability  reviews  (MM  and  PCR) . 

(m)  Production  readiness  reviews  (PRR) . 

(n)  Quality  or  product  assurance  assessments 
and  reviews. 

(o)  Any  other  areas  in  which  AFCMD  support 
would  be  available." 

The  above,  along  with  other  areas  in  AFSCR  800-42,  are 
essentially  spelled  out  in  detail  specific  for  software  in 
Table  Five.  Although  AFSCR  800-42  would  appear  to  imply  the 
increased  CAS  involvements  recommended  above,  interviews 
with  SPO,  AFPRO,  and  AFCMD  personnel  indicate  that  in  the 
software  area,  these  involvements  are  happening  infrequently 
or  not  at  all.  Assumedly,  it  is  the  prerogative  of  AFSC  to 
enforce  these  involvements. 
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7.0  CAS  PERSONNEL  SKILLS  AND  STAFFING 


The  third  major  requirement  of  this  phase  of  the  study 
was  to  utilize  the  technical  tasks  to  be  performed  by  the 
CAS  to  identify  personnel  skills  required  within  the  CAS 
organizational  functions  to  carry  out  those  tasks.  This 
corresponds  to  Task  4.3.3  of  the  SOW  for  the  study. 

As  a  preliminary  to  identifying  personnel  skills  to 
carry  out  tasks  which  the  prior  analyses  show  to  be 
prominent  for  CAS,  an  analysis  of  CAS  responsibilities  with 
regard  to  functional  areas  (Engineering,  Quality  Assurance, 
Manufacturing,  Contracting,  Program  Management  Support, 
Property  Management,  and  Subcontracting)  was  carried  out. 
Criterion  for  the  inclusion  of  a  task  in  this  analysis  was 
that  it  have  either  a  I  or  II  rating  in  the  analysis  shown 
in  Table  Five.  The  idea  for  this  criterion  was  to  provide  a 
first  approximation  of  those  tasks  where  CAS  personnel  have 
more  critical  responsibility.  A  table  showing  the  plot  of 
selected  tasks  against  responsibilities  assigned  each 
functional  area  is  included  in  Appendix  C.  Visual 
inspection  of  this  table  led  to  the  grouping  of  tasks  into 
two  areas: 

(1)  Those  tasks  where  high  level  of  responsibility  was 
spread  over  several  functional  areas. 

Those  tasks  where  high  levels  of  responsibility 
were  assigned  to  few  functional  areas.  In  the 


(2) 
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case  of  all  but  one  such  task,  the  only  two 
functional  areas  with  high  levels  of 
responsibility  were  Engineering  and  Quality 
Assurance. 

The  following  tasks  were  identified  as  having 
responsibility  spread  over  several  functional  areas: 

Task  Number  (refer  to  Appendix  C) 

15 

18 

19 

21 

23 

48 

49 

50 

52 

53 

All  of  the  remaining  tasks  are  evaluated  as  having 
higher  levels  of  responsibility  limited  to  few  functional 
areas . 

Since  the  tasks  representing  higher  responsibilities 
limited  to  few  functional  areas  included  (virtually)  only 
Engineering  and  Quality  Assurance,  the  skills  required  to 
perform  the  tasks  must  reflect  (1)  a  general  area,  (2)  an 
Engineering  area,  and  (3)  a  Quality  Assurance  area.  Tasks 
which  had  high  responsibility  rankings  only  for  Engineering 
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and  Quality  Assurance  are  shown  separately  for  these  two 
functional  areas  in  tables  in  Appendix  C.  These  tables 
match  tasks  for  Engineering  and  Quality  Assurance  against 
suggested  training  areas.  These  training  areas  are 
discussed  below. 

For  the  purposes  of  the  present  analysis,  the  tasks 
shown  in  Table  Five  as  requiring  major  CAS  input  only 
suggest  a  partitioning  of  CAS  functional  units  into  the 
three  following  categories: 

(1)  Engineering 

(2)  Quality  Assurance 

(3)  All  other  functional  units. 

The  implication  of  this  breakdown  is  that  Engineering 
and  Quality  Assurance  appear  to  have  the  most  general, 
specific,  long  duration,  and  continuous  contact  with  the 
development  of  software.  Other  units  tend  to  have  more 
general,  briefer,  or  more  technically  limited  contact  with 
software  development.  It  is  admitted  at  the  outset  that  any 
statements  of  this  nature  are  clearly  generalizations,  and 
there  are  likely  to  be  important  exceptions.  The  exceptions 
should  be  responded  to  in  accordance  with  their  specific 
training  needs,  whatever  those  should  be. 

The  information  in  Table  Five  and  the  three  categories 
just  outlined  above  lead  to  describing  three  areas  of 
training  for  CAS  personnel  (the  term  "training"  is  used  for 
convenience  here  to  refer  to  any  combination  of  formal  or 
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informal  training  and/or  experience  which  results  in  the 
knowledge/ ski 11  areas  outlined) : 

(1)  Training  of  a  basic  and  general  nature  which 
should  be  obtained  by  all  affected  CAS  personnel 
(includes  Engineering  and  Quality  Assurance) . 

This  area  is  hereafter  referred  to  as  —  Software 
Development  Management  Evaluation. 

(2)  Specific  training  areas  relevant  to  affected 
Engineering  personnel.  This  area  is  hereafter 
referred  to  as  —  Software  Development  Technical 
Engineering  Evaluation. 

(3)  Specific  training  areas  relevant  to  affected 
Quality  Assurance  personnel.  This  area  is 
hereafter  referred  to  as  —  Software  Quality 
Assurance  Evaluation. 

The  Software  Development  Management  Evaluation  Area  is 
intended  to  give  a  general  and  basic  background  in  software 
acquisition  for  all  affected  CAS  personnel.  This  broad  area 
is  seen  as  probably  sufficient  coverage  of  the  area  for  all 
personnel  except  for  Engineering  and  Quality  Assurance.  In 
the  case  of  Engineering  and  Quality  Assurance  personnel, 
Software  Development  Management  Evaluation  is  seen  as 


introductory  and  basic  to  the  other  areas  more  specific  to 
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Engineering  and  Quality  Assurance.  One  advantage  of 
identifying  an  area  such  as  this  is  that  it  establishes  a 
"core"  and  a  common  basis  of  knowledge,  understanding,  and 
communication  for  all  CAS  personnel  involved  with  software 
acquisition. 

The  more  specific  training  areas  under  the  label  of 
Software  Development  Management  Evaluation  are: 

-  Acquisition  and  Development  Cycles  for  Software 

-  Cost  Estimation  Methods  and  Problems  in  Software 
Development 

-  Management  Principles  in  Software  Development 

-  Quality  Factors  throughout  the  Software  Life  Cycle 

-  Cost  Factors  throughout  the  Software  Life  Cycle 

-  General  Requirements  and  Intentions  of  MIL-STD-2 167 . 

Engineering  Training.  Using  the  background  described  above 
as  a  prerequisite  foundation,  specific  areas  of  training  for 
Engineering  are  grouped  under  the  heading,  Software 
Development  Technical  Engineering  Evaluation: 

-  Technical  Cost  Aspects  of  Software  Planning 

-  Technical  and  Quality  Factors  in  Requirements 
Analysis 

-  Software  Specification  Techniques 

-  Engineering  Management  in  the  Software  Life  Cycle 

-  Software  System  Development  Techniques 

-  Software  Test  Development  Techniques 
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-  Software  Program  Design  and  Coding 

-  All  of  the  above  are  assumed  to  involve  MIL-STD-2167 
and  current  or  future  regulations  and  additional 
standards  as  appropriate. 

Quality  Assurance  Training.  At  the  outset,  it  must  be 
commented  upon  that  Quality  Assurance  has  a  system  (called 
"KSAs"  for  Knowledges,  Skills,  and  Abilities)  which  arises 
from  and/or  is  linked  to  job-task  analysis.  The  KSA  system 
is  very  advantageous  for  analyzing  and  planning  training 
needs.  The  analysis  presented  here  is  based  on  broad  tasks, 
whereas  the  KSAs  are  based  on  a  much  more  detailed  job-level 
task  orientation.  The  data  presented  here  may  possibly  shed 
light  on  KSAs,  but  should  in  no  way  be  seen  as  an  approach 
to  critique  or  change  the  KSA  system. 

The  following  areas  of  software  training  are  assumed  to 
rest  on  a  foundation  of  the  knowledge  in  the  Software 
Development  Management  Evaluation,  or  core  area,  and  are 
here  labelled  Software  Quality  Assurance  Evaluation: 

-  Software  QA  plans  and  Software  QA  plan  development 

-  Software  QA  in  test  plan  auditing  and  test  execution 
auditing  throughout  the  Software  Life  Cycle 

-  Software  configuration  management  auditing  throughout 
the  Software  Life  Cycle 

-  Software  Quality  Assurance  standards  for  software 
development  throughout  the  Life  Cycle 
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-  All  of  the  above  are  assumed  to  involve  MIL-STD-2167 
and  current  or  future  relevant  regulations  and 
standards. 

Further  Recommendations  for  Training  Development.  The  areas 
outlined  here  arise  clearly  from  the  task  analyses  which 
comprise  this  study.  A  next  step  in  the  production  of  high- 
utility  training  would  be  to  utilize  the  areas  identified 
here  as  guidelines  to  conduct  more  detailed  job-task 
analyses  and  arrive  at  highly  specific  training  objectives 
such  as  those  currently  found  in  the  Quality  Assurance  KSAs. 

It  is  recommended  that  all  CAS  training  implied  here  be 
refined  to  the  level  of  specificity  exemplified  by  the  KSAs. 
Such  systems  of  specific  training  objectives  offer  a  number 
of  benefits,  such  as: 

(1)  A  basis  to  evaluate  the  comprehensiveness  of 
training  considered 

(2)  A  system  for  continuous  evaluation  of  specific 
training  needs  of  individuals 

(3)  A  system  for  determining  needs  of  groups 

(4)  A  system  for  establishing  priorities  for  training 
when  distributing  limited  resources 

(5)  A  system  for  tracking  specifically  where  training 
funds,  resources,  and  time  have  been  applied 

(6)  A  system  to  clarify  training  needs  requested  from 


training  contractors. 
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The  net  result  of  these  advantages  should  be  more 
effective  training  and  more  cost-effective  training.  As  an 
example,  the  Quality  Assurance  KSAs  are  included  in  an 
Appendix  to  this  report  in  their  entirety. 

The  foregoing  discussion  is  oriented  exclusively  to 
recommendations  for  the  type  and  content  of  training 
required  by  the  various  CAS  organizations.  Another 
important  aspect  of  the  topic  of  CAS  software  skills  is  the 
availability  of  manpower  resources. 

Interviews  with  CAS  personnel  at  AFPRO  sites  and  at 
HQ/AFCMD  consistently  reflected  an  overall  shortage  of 
manpower  in  addition  to  a  shortage  of  personnel  adequately 
skilled  in  software.  The  prevailing  point  of  view  might  be 
characterized  by  the  rationale  that  since  you  have  too  few 
personnel  to  adequately  do  the  parts  (nonsoftware)  of  the 
job  you're  best  prepared  for,  it  doesn't  make  much  sense  to 
push  hard  into  software,  where  you  know  you  have  an  extreme 
shortage  of  skills. 

The  Electronics  Institute  of  America  (E.I.A.)  estimates 
the  total  USAF  personnel  need  in  1986  at  8,800  people  for 
software  support  (which  includes  CAS  personnel) .  The  E.I.A. 
estimates  that  this  requirement  will  increase  at  an  average 
rate  of  14%  per  year  for  the  next  three  years.  Given  the 
widely  acknowledge  observation  that  CAS  software  is 
currently  undermanned,  the  emphasis  required  to  overcome  the 
current  deficit  and  also  keep  up  with  an  average  14%  per 


Dennis  Blosser  &  Associates 
Final  Report 
Page  104 

year  growth  rate  (assuming  proportionality  of  CAS  to  other 
software  support  personnel)  would  be  very  weighty.  In  fact, 
given  the  current  pressures  to  cut  back  government 
expenditures  due  to  Gramm-Rudman,  it  seems  very  unlikely 
that  the  necessary  growth  rate  could  be  expected  even  if 
there  were  not  the  initial  deficit  to  surmount. 

Although  cut-backs  due  to  Gramm-Rudman  have  not  yet 
seriously  eroded  the  number  of  current  CAS  software 
personnel,  the  preceding  discussion  makes  it  clear  that  for 
this  area,  simply  holding  the  status  quo  actually  means 
losing  ground  at  a  rapid  rate.  Thus,  if  the  future  effects 
of  Gramm-Rudman  on  other  policies  result  in  zero  or  slow 
growth  of  CAS  software  personnel,  the  current  manpower 
shortage  will  quickly  worsen. 

Another  critical  factor  for  CAS  personnel  skills  is 
verticality  of  training  in  software.  Availability  of  more 
and  better  trained  personnel  at  the  journeyman  level  will 
not  effectively  impact  USAF’s  needs  for  CAS  support  in 
software  acquisition  unless  there  is  a  pervasive  elevation 
of  the  organizational  visibility  and  importance  of  software. 
This  can  come  about  only  through  training  of  all  levels  of 
management,  from  the  very  top  through  supervision.  The 
concern  and  attention  that  management  gives  to  the  various 
functional  areas  of  an  organization  affects  the  degree  to 
which  people  involve  themselves  in  that  work,  and  how 
effectively  they  do  that  work,  independent  of  their 
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training.  Management  at  all  levels  can  give  the  appropriate 
type  and  degree  of  emphasis  to  software  only  with  sufficient 
training. 

In  closing,  some  comments  should  be  made  regarding  the 
current  status  of  software  training  in  Engineering  and 
Quality  Assurance.  As  stated  above,  Quality  Assurance  has  a 
powerful  base  for  their  journeyman  training  program  in  their 
system  of  KSAs.  This  is  especially  important  since  hiring 
in  this  area  is  based  on  the  assumption  that  incoming 
personnel  with  general  or  limited  expertise  will  be  hired 
and  then  continuously  assessed  and  trained  to  expand  and 
deepen  their  capabilities.  Currently,  software  is  a  top 
priority  for  journeyman  training.  The  big  catch  is  funding. 
In  prior  years,  this  training  program  was  well-funded  and 
delivered  many  hours  of  specifically  targeted  training  for 
its  journeymen.  However,  for  the  last  three  years,  funds 
allocated  for  this  training  have  been  cut  so  low  that  they 
represent  a  virtual  zero.  So  here  is  a  program  geared  by 
need,  premise,  and  capability  that  is  severely  impaired  by 
recent  lack  of  funding.  (Quality  Assurance  also  operates 
three-year  training  intern  programs  which  are  not  considered 
in  this  discussion  which  is  targeted  to  journeyman 
capabilities . ) 

Lack  of  funds  is  also  a  currently  critical  factor  for 
training  in  Engineering.  In  Engineering,  the  hiring  premise 
differs  from  Quality  Assurance  in  that  the  incoming  employee 
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is  a  professional  engineer  by  virtue  of  training  and 
experience.  However,  CAS  engineers,  unlike  civilian 
engineers,  are  expected  to  become  generalists  and  grow  to 
operate  effectively  in  more  areas  than  the  specialty  of 
their  training  or  prior  experience.  MCCR  personnel  at 
HQ/AFCMD  are  well  aware  that  training  engineers  from  other 
areas  in  software  is  a  necessary  source  for  developing 
manpower  capability  in  this  area.  Although  their  training 
program  is  not  developed  to  the  level  of  specificity  and 
delivery  capability  of  the  current  Quality  Assurance 
program,  Engineering  aggressively  studies  their  needs  for 
training.  Currently,  lack  of  funds  have  prevented  them  from 
mounting  a  systematic  program.  Currently,  most  of  their 
training  is  provided  by  HQ  staff  or  administered  by  them  via 
circulation  of  training  videotapes  to  the  AFPROs. 
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8.0  RECOMMENDATIONS  REGARDING  POLICY 

The  fourth  requirement  for  this  phase  of  the  study  was 
to  recommend  policy  changes  that  would  provide  clear  and 
concise  functional  tasking  for  MCCR  acquisition.  This 
corresponds  to  Task  4.3.4  of  the  SOW  for  the  study. 

The  problems  and  limitations  involved  in  attempting  to 
produce  completely  clear  and  concise  separation  of 
functional  tasks  for  SPO  and  CAS  have  been  discussed  in 
Chapter  6.0  of  this  report  To  reiterate,  as  appealing  as 
complete  separation  of  function  may  appear  from  a  purely 
administrative  viewpoint,  oversimplification  threatens  the 
functional  effectiveness  of  the  complex  process  of  software 
acquisition. 

Thus,  the  task  relations  specified  in  Table  Five  are 
designated  as  recommendations  for  policy,  and  it  is 
recommended  that  policy  should  be  altered  to  enforce  the 
relationships  of  Table  Five.  This  contractor  is  aware  that 
some  of  the  recommendations  following  Table  Five  and  many  of 
the  relationships  shown  in  Table  Five  already  exist  in 
written  policy  as  "suggested,"  "where  appropriate"  (or 
similar  phrases)  or  are  even  required  (e.g.,  AFSCR-800-42) . 
However,  interviews  conducted  in  this  study  show  that  in 
spite  of  guidance  or  requirements  of  written  policy,  in 
practice,  there  is  insufficient  involvement  of  CAS 
capabilities  in  the  development  of  RFPs  and  contracts  and 
too  little  early  involvement  of  CAS  personnel  in  formal 
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reviews  and  audits  to  fully  utilize  and  support  their 
potential . 

Therefore,  policy  should  be  rewritten  to  require  and 
enforce  CAS  involvement  as  shown  in  Table  Five  and  discussed 
in  Chapter  6.0  of  this  report. 

The  SOW  for  this  study  requests  changes  be  recommended 
"to  DOD  policy,  including  the  Federal  Acquisition  Regulation 
(FAR) ,  the  DOD  FAR  Supplement  (Sup) ,  the  Air  Force  FAR  Sup, 
and  the  AFSC  FAR  Sup....”  Recommendations  are  made  as 
indicated  above.  However,  they  are  tentative.  The 
contractor  would  deem  it  precipitous  to  institute  such 
sweeping  changes  in  policy  without  further  study. 

Thus,  it  is  further  recommended  that  policy 
requirements  and  enforcements  such  as  those  recommended 
above  be  tested  on  several  projects,  and  the  results 
evaluated  prior  to  instituting  more  widespread  or  higher 
level  changes  in  policy. 
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9.0  ANCILLARY  RECOMMENDATIONS 

As  in  any  study,  there  was,  of  course,  more  information 
observed  than  is  strictly  relevant  to  the  focal  issues. 

Some  additional  recommendations  (or  perhaps  substitute 
"suggestions"  for  "recommendations,"  if  preferable)  are 
included  here. 

There  are  many  interesting  findings,  but  the  most 
salient  for  additional  recommendations  are  informally: 

(a)  There  are  some  significant  overlaps  of  function  in 
SPO  and  AFCMD  activities  for  the  acquisition  of 
MCCR  software.  However,  these  appear  to  be  well- 
noted  by  the  individuals  involved  in  actual 
operations . 

(b)  The  primary  formal  instrument  for  coordination  of 
activities  and  responsibilities  between  the  SPO 
and  AFCMD  for  a  given  contractor  operation  is 
usually  referred  to  as  the  MOA  (memorandum  of 
agreement) .  This  instrument  is  stipulated  by 
policy,  but  it  is  admirably  flexible.  In  most 
cases,  however,  it  is  reported  as  minimally 

functional . 

(c)  Generally,  SPO  personnel  involved  in  software 
activities  are  spread  very  thin,  feel  very 
overloaded  and  believe  that  AFCMD  personnel 
contribute  little  or  nothing  to  MCCR  software 
acquisition. 


/ 
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(d)  Generally,  AFCMD  personnel  believe  their 
capabilities  and  contributions  to  MCCR  software 
acquisition  are  ignored/under-utilized  by  SPOs. 

(e)  There  are  a  few,  but  notable  exceptions  to  (c)  and 
(d)  above,  which  point  the  way  to  positive 
recommendations . 

(f)  There  is  a  lack  of  sufficiently  trained  and 
experienced  personnel  in  both  SPOs  and  AFCMD  to 
handle  the  extremely  rapidly  growing  requirements 
of  the  MCCR  software  acquisition  process. 

Ancillary  recommendations  emerging  from  the  findings  of 
this  study  offer  the  possibility  (but  not  the  guarantee)  of 
more  effective  operations  by  SPOs  and  AFCMD  in  MCCR  software 
acquisition.  Such  increased  effectiveness  could  lead  to 
better  cost  and  schedule  control  and  the  increased  assurance 
of  receiving  quality-controlled  software  products. 

Unfortunately,  the  ancillary  recommendations  of  this 
study  would  require  considerable  commitment  and  yet  address 
only  a  segment  of  the  larger  and  complex  overall  problem  of 
MCCR  software  acquisition. 

The  problem  of  time  pressure  discussed  in  Section  5.1 
pervades  every  aspect  of  MCCR  software  acquisition  and  also 
affects  the  other  two  root  problems  of  insufficient 
trained/experienced  personnel  and  under-utilization  of 
available  capability  in  AFCMD.  Specifically,  when  a  work 
force  is  under  heavy  time  pressure,  it  is  unlikely  to  find 
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adequate  time  to  identify  training  needs,  develop  training 
plans,  and  find  available  time  to  release  personnel  for 
training.  Further,  when  an  organization  such  as  an  SPO  is 
under  heavy  time  pressure,  it  is  not  likely  to  find  time  to 
develop  or  encourage  uncontrolled  critical  input  from  an 
outside  organization  such  as  AFCMD.  Some  of  the  likely 
fundamental  causes  of  time  pressure,  such  as  the 
difficulties  of  accurately  estimating  the  time/cost 
parameters  of  software  production  scheduling  and  the 
difficulties  of  planning  and  executing  strategic  weapons 
development  in  a  rapidly  changing  technology,  are  well 
beyond  the  scope  of  this  study. 

However,  there  is  a  reciprocal  relationship  between  the 
time-pressure  problem  source  and  the  other  two  fundamental 
problems.  That  is,  if  means  could  be  identified  to  enhance 
the  utilization  of  AFCMD  software  capability  and  also  to 
increase  the  skill  levels  of  personnel  involved  in  MCCR 
software  acquisition  with  minimal  time  requirements,  these 
two  problem  areas  would  be  improved  and  would  have  the 
potential  to  reduce  at  least  some  time  pressure.  Thus,  the 
following  recommendations: 

1.  Service-Marketing  Approach  for  AFCMD.  In  the  majority 
of  cases  where  SPOs  seem  to  make  little  use  of  AFCMD 
capabilities,  there  is  also  a  benign  ignorance  typified  by 
statements  such  as,  "I  don't  know  what  they  really  do,"  or, 
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"I  suppose  they  could  help  us,  but  they  report  to  In 

the  case  of  AFCMD  personnel,  the  typical  attitude  is,  "We 
could  offer  the  SPOs  a  lot  of  information  that  would  help 
them  and  maybe  lead  to  better  software,  but  they're  not 
interested. " 

This  is  particularly  noteworthy  in  that  the  policy  for 
both  organizations  specifically  acknowledges  benefits  of 
AFCMD  expertise  for  SPO  operations,  and  MOAs  are  prepared  to 
specify  and  assure  the  ways  the  organizations  will  recognize 
and  support  each  other  in  individual  projects. 

In  the  apparent  minority  of  cases  where  SPOs  and  AFCMD 
organizations  have  productive,  effective  relationships,  the 
situation  is  characterized  by  the  following  key  elements: 

(a)  A  relationship  was  established  by  initiative  from 
AFCMD . 

(b)  The  major  thrust  of  this  initiative  came  from  a 
high-ranking  (e.g.,  EP  division  chief)  in  an  AFCMD 
in-plant  organization. 

(c)  The  effort  was  persuasive  in  nature,  based  on  how 
AFCMD  can  benefit  the  SPO  and  how  as  a  team  AFCMD 
and  SPO  can  influence  the  contractor  to  produce 
better  software. 

(d)  The  effort  was  continued  over  a  period  of  time 
(i.e.,  not  a  one  or  two  presentation  information 
offering,  for  example) . 
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Observation  of  these  factors  led  to  the  following 
recommendation  of  what  is  being  termed  here  a  "service¬ 
marketing  approach"  for  AFCMD  in  MCCR  software  acquisition. 
The  main  objective  of  this  recommended  approach  is  to 
increase  SPO  utilization  of  AFCMD  capabilities.  The  basic 
features  of  this  recommendat ion  are  as  follows: 

(a)  Specify  exactly  what  services  AFCMD  wishes  to 
market  (in  general)  to  SPOs. 

(b)  Determine  specifically  what  actions  by  an  SPO  do/ 
do  not  constitute  acceptance  of  each  possible 
component  of  the  service  product. 

(c)  Determine  specifically  what  benefits  each  service 
product  offers  SPO  and  under  what  conditions. 

(d)  Determine  specifically  who  in  SPO  or  related 
organizations  makes  the  decision,  and  when,  to 
accept  AFCMD  service  products  or  not. 

(e)  Set  specific  goals  for  level  of  acceptance  of 
service  products. 

(f)  Set  specific  goals  for  activities  to  obtain 
acceptance  of  service  products. 

(g)  Evaluate  and  adjust  (e)  and  (f)  on  experience. 

(h)  The  activities  of  (f)  constitute  persuasive, 
repeated,  timely  demonstrations  to  decision  makers 
which: 
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(1)  clearly  and  specifically  show  benefits  to  SPOs 
and  USAF. 

(2)  include  a  mechanism  for  commitment. 

(i)  These  activities  need  to  be  continuous  and  long 
term  in  duration. 

(j)  A  component  of  public  relations  should  also  be 
developed,  along  with  seeking  support  from  a  wide 
and  vertical  constituency. 

2.  Streamlined  training.  In  an  age  of  rapid  technological 
turnover  (especially  true  in  the  software  area)  and  with  the 
personnel  turnover  and  reassignment  that  is  prevalent  in 
government  organizations,  streamlining  of  training  is  very 
important.  The  government  has  courses  available  in  software 
acquisition  management.  Although  these  could  no  doubt  be 
improved,  they  appear  to  offer  an  adequate  initial  basis  of 
training  in  the  area.  However,  an  observation  of  this  study 
is  that  relatively  few  have  been  able  to  take  these  courses. 
Currently  these  courses  are  taught  primarily  at  training 
centers.  Perhaps  less  lengthy,  videotaped,  or  otherwise 
mediated  courses  could  be  developed  which  could  be 
disseminated  more  widely  and  used  more  time  effectively. 

As  a  follow-up  to  the  basic  or  foundational  training, 
it  is  recommended  that  all  more  advanced  training  be 
comprised  in  a  computerized  dynamic  data  base.  This  data 
base  could  utilize  an  artificial  intelligence  language  such 
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as  PROLOG.  Persons  needing  training  beyond  the  basics  of 
software  acquisition  management  could  simply  query  this  data 
base.  The  data  base  could  be  continuously  updated  by 
experts  from  within  AFCMD  and  external  to  AFCMD.  This 
recommendation  reduces  the  training  requirements  to  two 
elements : 

(1)  Basic  software  acquisition  management. 

(2)  How  to  utilize  the  training  data  base  for  advanced 


training  and  updating. 


The  contractor  is  aware  that  other  proposals  and 
efforts  have  been  made  in  AFCMD  that  have  been  associated 
with  the  idea  of  ’’marketing. "  It  should  be  noted  that  the 
approach  suggested  here  is  very  specific  and  goal  oriented 
and  includes,  but  is  not  centered  around,  an  image-promotion 
approach.  Image  promotion  is  useful  and  can  be  effective  up 
to  a  point.  However,  it  is  difficult  or  impossible  to  gauge 
the  effectiveness  of  such  a  program  and  thus  to  effectively 
modify  it  or  make  other  decisions.  The  program  outlined 
here  supports  direct  evaluation  and  decision  making. 

Assumedly,  leadership  and  direction  for  a  service¬ 
marketing  effort  would  come  from  Headquarters,  AFCMD.  The 
main  thrust  for  effectiveness,  though,  has  to  come  from  the 
in-plant  AFCMD  organizations.  In  the  final  analysis  it  is 
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the  perceived  capability  of  these  organizations  that  SPOs 
"buy"  or  not. 

A  major  question  for  the  streamlined  training 
recommendation  is,  "Who  would  be  in  charge  of  the  data 
base/experts?"  This  is  a  political  question  beyond  the 
scope  of  this  study.  However,  at  a  minimum,  the  following 
need  to  be  included.  First,  in  spite  of  a  shortage  of 
adequately  trained/experienced  personnel  in  the  field,  there 
are  some  very  competent,  dedicated  people  in  the  AFCMD  in- 
plant  organizations.  Their  capabilities  should  be  very 
valuable  here.  Second,  some  specific  mechanism  should  be 
established  to  assure  that  the  content  and  function  of  the 
data  base  continuously  serves  the  intended  end  users  well. 
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Foreward 


While  much  attention  has  been  placed  on  the  life  cycle  of  embedded 
computer  systems,  somewhat  less  attention  tends  to  be  directed  to  those 
computer  resources  supporting  the  delivery  and  maintenance  of  end  products. 
Many  lessons  have  been  learned  by  painful  experiences  in  acquiring  and  using 
computer  systems  for  military  weapon  systems.  Now  the  computer  is  finding 
its  role  in  a  wide  range  of  applications,  many  directly  affecting  the 
design,  configuration,  manufacturing,  testing,  and  inspection  of  contracted 
products.  Some  of  these  computer  applications  are  acquired  from  vendors, 
some  adopted  from  existing  or  like  systems  and  still  others  are  custom 
built.  Regardless  of  its  source  of  being,  computer  resources  used  in  the 
work  place  will  need  some  degree  of  management.  Usually,  the  computer 
programs  become  the  focus  for  management,  as  they  are  invariably  the  scape 
goat  of  user  complaints. 

So  how  should  one  approach  the  managment  question  of  in-plant  support¬ 
ing  computer  resources?  How  can  we  avoid  management  overkill  or  costly 
decisions?  Simplistically  one  could  say,  "Well,  use  the  basic  principles 
applied  for  deliverable  product^"  or  “What  is  good  for  the  goose  is  good  for 
the  gander."  Perhaps  you  have  your  own  trite  statement  to  insert  here.  In 
any  event,  this  guide  makes  an  attempt  to  bridge  the  gap  between  management 
practices  for  deliverable  computer  resources  and  doing  nothing.  We  have 
attempted  to  address  practical  reasons  for  adopting  certain  management 
approaches.  And,  yes,  even  a  touch  of  lessons  learned  has  been  thrown  in. 
Management  of  any  kind  of  computer  resource  is  important,  regardless  of  its 
application,  this  perspective  of  broad,  non  categorized  CR  management  is 
presented.  We  are  hopeful  that  this  guide  will  provide  the  framework  you 
need  and  can  use. 
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1.0  GUIDE  OBJECTIVES:  This  guide  was  prepared  to  assist  managers  structure 
and  understand  policies  and  procedures  in  the  management  of  nondeliverable 
computer  resources  ( NDCR ) .  It  is  the  intent  of  this  guide  to  capture  the 
basic  management  methods  and  lessons  learned  attributed  to  deliverable 
embedded  computer  systems  and  translate  them  into  management  terms  appro¬ 
priate  for  NDCR. 

1.1  Applicability:  This  guide  should  applied  to  any  NDCR,  i  .e., 
hardware,  software,  and  associated  documentation,  used  by  a  manager's 
organization  to  plan,  evaluate  or  produce  a  deliverable  product.  It  is 
intended  that  the  reader  compare  his/her  NDCR  environment  to  this  guide  and 
tailor  one’s  own  policies  and  procedures  according  to  the  management  risks 
and  disciplines  involved.  The  guide  user  may  identify  NDCR  used  in  their 
environment  that  do  not  exactly  fit  any  categorization  addressed  by  this 
guide.  This  should  not  be  an  inhibitor  to  using  the  guide.  Not  every  idea 
or  consideration  will  apply,  but  concepts  addressed  an  initial  step  in 
establishing  or  refining  NDCR  management  policies  and  procedures. 

1.2  Need  Recognition:  Do  you  need  the*  guide?  Take  a  moment  to 
evaluate  your  environment,  i.e.,  look  at  how  computers  and  computer  programs 
are  being  assembled  and  are  used  to  aid  your  job.  Oo  you  know  how  any  one 
computer  system  or  computer  program  affects  your  operations  and  to  what 
degree?  If  you  cannot  confidently  respond  to  this  question,  then  we  suggest 
you  proceed  into  this  guide.  NDCRs  possess  risks  to  your  working  environ¬ 
ment  even  if  the  computer  system  salesman  did  not  point  out  such  facts.  It 
is  very  easy  to  be  absorbed  into  the  computer  technology  boom  which  makes  it 
just  as  easy  to  make  unfortunate  management  errors .  This  quicifc attempts  to 
raise  many  environmental  conditions  for  your  consideration  and  proposes  some 
guidelines  to  deal  with  those  conditions. 

2.0  DEFINITIONS:  Technical  terms  used  in  this  guide  are  consistent  with 
IEEE  Standard  729,  "A  Glossary  of  Software  Engineering  Terminology." 
Defined  terms  not  contained  in  that  standard  are  listed  below. 

2.1  Nondeliverable  Computer  Resources  (NDCR):  Resources,  i.e., 

hardware,  software  an3  associated  documentation,  which  are  purchased, 
developed  or  used  in  the  design,  development,  manufacturing,  inspection,  or 
test  of  a  product  deliverable  to  a  customer. 

3.0  MANAGEMENT  OBJECTIVES:  Nondeliverable  Computer  Resources  (NDCR) 

represent  a  growing  investment  by  management.  Management  will  rely  more  and 
more  on  NDCR  to  assist  every  element  of  the  work  force  to  better  support  the 
systems  acquisition  process.  However,  with  the  gains  in  productivity  have 
come  new  risks  that  threaten  the  cost  and  schedule  savings.  To  safeguard 
the  integrity  of  NDCRs,  a  series  of  management  objectives  is  identified  and 
discussed  in  the  following  sections. 

One  word  of  caution;  tailoring  management  objectives  is  extremely  important 
for  NDCR.  One  should  avoid  creating  elaborate  methodologies  or  bureaucra¬ 
cies  to  manage  NDCR.  The  key  word  is  elaborate#  Methodologies  or  bureaucra¬ 
cies  will  generally  be  needed  to  assure  that  workable  discipline  is  formu¬ 
lated  and  used.  In  some  instances,  management  may  desire  to  distinuish 
between  NDCR  systems,  e.g.,  ATE  vs.  CAD,  to  define  different  management 
methodologies.  This  maybe  a  form  of  typing  or  categorizing  NDCR  in  a  way  to 
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avoid  the  use  of  stringent  controls  w  very  formal  procedures  where  simple, 
informal  methods  would  suffice. 


3.1  PI anninq:  Project  planning  should  include  detailed  consideration 
for  the  development,  purchasing,  control,  evaluation,  validation  and  use  of 
nondeliverable  computer  resource  products. 

3.1.1  General:  Where  NDCR  will  be  developed/acquired  by  a  system 
developer,  specific  identification  of  schedules,  labor  and  computer  time, 
and  activity  interfaces  for  each  NDCR  system  should  be  documented.  Project 
and  customer  management  need  to  be  aware  of  potential  risks  and  benefits  to 
the  project  by  engaging  in  parallel  computer  resource  developments/acquisi¬ 
tions  . 


3.1.2  Interfaces:  It  is  not  uncommon  for  NDCR  to  depend  on 
events  outside  of  NDCR  management  control.  NDCR  usually  will  support  a 
major  project  and,  thereby,  will  suffer  from  changes  dictated  by  the 
project.  The  NDCR  manager  can  expect  some  suffering,  but  only  to  the 
degree  that  internal  interfaces  have  not  been  established  and  effectively 
cultivated.  The  internal  project  change  control  and  appropriate  design 
review  forums  should  be  examined  closely  for  potential  direct  interfaces. 
The  decisions  arising  from  these  forums,  in  some  cases,  are  still  too  late 
for  the  NDCR  manager  to  adequately  support  project  management,  but  it  is  a 
lot  closer  than  waiting  for  formally  published  minutes,  design  change 
orders  or  other  communiques. 

3.1.3  .Skills:  Applying  computer  technology  to  a  manual -labor 
task  may  be  traumatic.  It  is  not  uncommon  for  management  to  take  a  task 
expert,  e.g.,  an  RF-engineer,  and  train  that  person  how  to  program  a 
computer,  e.g.,  using  8ASIC.  This  may  seem  quite  innocent  and  even  log¬ 
ical,  but  the  NDCR  manager  should  beware  of  a  major  pitfall.  If  software 
or  complex  data-bus  interfaces  will  be  developed,  the  NDCR  manager  should 
seek  computer  system  technical  skills  to  supplement  a  staff  of  traditional 
electronics,  manufacturing,  structure,  etc.,  skills.  It  is  rare  to  have 
multi-qualified  individuals  who  know  the  total  computer  system  technically 
and  have  the  discipline  to  get  the  job  done.  Discipline  is  the  key.  The 
methodology  followed  by  a  software  designer  must  be  highly  disciplined  to 
assure  that  the  end  product  can  be  maintained  and  documented.  It  is  not 
uncomnon  for  a  task  expert  (i  .e.,  an  electronic  engineer,  manufacturing 
specialist,  etc.)  not  to  receive  the  training  on  how  to  do  efficient  and 
effective  software  design  and  documentation.  Hence,  a  “team"  approach 
involving  several  disciplines  can  be  most  beneficial  to  the  NDCR  manager. 

3.1.4  Schedules:  Developing  a  computer  system  (or  NDCR)  to 
support  some  phase  of  a  deliverable  product  development  or  acceptance  <*?n 
be  very  sporting.  Management  attempts  to  define  a  realistic  schedule  for 
the  NDCR  can  be  easy.  Using  and  maintaining  the  schedule  can  be  a  real 
challenge.  As  discussed  before,  the  NDCR  manager  will  find  that  various 
interfaces  with  the  deliverable  product  management  will  be  essential. 
Relating  as  many  key  del iverbl e-product  related  milestones  to  the  NDCR 
being  developed  is  critical.  Not  only  are  interfaces  better  defined,  but 
the  NDCR  manager  can  better  assess  the  success  of  the  NDCR  development  when 
tied  directly  to  the  deliverable  product  and  its  problems. 
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As  with  large  computer  systems,  software  development,  if 
any,  will  typically  pace  the  NDCR  development.  The  "team"  of  skills  doing 
the  NDCR  software  development  and  the  "firmness"  of  NOCR  system  require¬ 
ments,  will  probably  drive  the  NDCR  software  development.  However,  take 
note  of  other  factors  as  security,  safety,  and  state-of-the-art  controlled 
equipment.  In  some  situations,  these  factors  may  be  very  significant.  The 
NDCR  manager  should  not  underestimate  the  risks  and  schedule  realities 

associated  with  these  factors.  Upper  management  commonly  fails  to  under¬ 
stand  the  complexity  of  automating  any  process.  Automation  may  be  seen 

only  from  a  hardware  or  electronics  point  of  view,  i.e.,  simple  electronic 

or  electrical  connections,  straight  forward  electronics  maintenance,  etc. 
This  perception  must  be  altered  to  provide  a  visible  picture  of  the  soft¬ 
ware  impact.  In  many  cases,  the  schedule  becomes  an  important  vehicle  to 
demonstrate  the  critical  hardware  and  software  factors.  Failure  to  realis¬ 
tically  account  for  these  factors  in  the  earliest  schedule  will  haunt  the 
project  manager  later  on. 

3.1.5  Return  on  Investment:  To  automate  for  the  sake  of  auto¬ 

mation  is  not  wise.  Although  personal  computers  have  made  automation  more 
feasible  and  less  costly,  management  still  needs  to  consider  the  "real" 

benefits  to  acquiring  computer  resources.  Why?  Because  many  factors  come 
into  play  in  supporting  and  using  the  NOCR.  Computer  system  salespersons 
are  very  quick  to  show  how  inexpensive  a  new  system  may  be.  Unfort.unatly , 
they  may  easily  gloss  over  many  hidden  costs  associated  with  expanding, 
interfacing,  updating  and  maintaining  the  computer  system  software,  hard¬ 
ware  and  documentation.  In  other  words,  look  at  any  computer  system  from  a 
life-cycle  point  of  view. 

3.1.6  System  Support:  Care  must  be  taken  by  the  NOCR  manager  to 
acquire  the  needed  documentation  or  vendor  support  agreements  for  the 
computer  system  used.  Again,  long  range  planning  should  address  the  simple 
aspect  of  the  computer  operating  system  maintenance  or  equipment  obso¬ 
lescence.  Knowing  how  long  a  vendor  will  truely  support  your  computer 
system  may  determine  your  strategy  to  acquire  specialized  in-house  support 
capabi 1 ity . 

3.2  Corrective  Action:  Corrective  action  procedures  should  be 
established  to  apply  to  nondeliverable  software  to  assure:  proper  identi¬ 
fication/documentation  of  software  problems;  analysis  of  problem  causes; 
validation  of  problem  solution;  authorization  to  implement  changes;  valida¬ 
tion  and/or  verification  of  change  implementation  correctness  and  complete¬ 
ness  . 


3.2.1  Approach:  Establishment  of  corrective  action  procedures 
should  reflect  a  structured  discipline  (i.e.,  controls,  review*  audits) 
commensurate  with  the  project  and  product  risks  of  using  any  huCR  system. 
The  discipline  established  by  the  developer/user  of  NDCR  should  be  a  common 
sense  sequence  of  actions.  These  actions,  as  described  in  the  following 
section,  should  form  a  closed-loop  process. 

3.2.2  Recording  Problems:  Establishing  a  written  record  of 
software  problems.  The  record  should  be  a  simple  logging  process  or  a  more 
formal  software  problem  record  form.  The  log  or  form  serves  as  a  reference 
point  for  management,  designers  and  programmers  alike  to  understand  the 
attention  that  a  problem  is  receiving. 
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3.2.3  Solutions:  With  any  problem,  some  record  of  the  proposed 
solution  and  an  analysis  of  its  implementation  is  needed.  This  is  the 
heart  of  the  corrective  action  process.  In  complex  applications,  this 
might  be  non-trivial;  many  interfaces  might  exist  and  management  needs  to 
know  the  cost /schedule/technical  implications  BEFORE  committing  to  a  change 
of  significance.  Part  of  this  step  involves  prel i mi  nary  steps  to  validate 
a  proposed  change  using  working  files  and  computer  programs.  These  results 
are  important  to  know  in  sufficient  detail  before  the  change  is  approved 
for  final  implementation. 

3.2.4  Authorization:  Authorizing  the  implementation  of  a  change 
is  a  very  important  step  in  any  corrective  action  process.  Whether  it  is  a 
change  generated  in-house  or  a  new  version  of  vendor  software,  appropriate 
management  and  technical  people  need  to  authorize  the  implementation. 
Sometimes  the  analysis  of  the  proposed  change  will  reveal  some  constraints 
that  management  might  not  wish  to  enjoin.  Delaying  a  change  to  a  latter 
time  might  prove  more  cost  effective  or  less  disruptive  to  an  ongoing 
process.  The  decision  needs  to  be  made  by  the  right  people.  It  should  not 
be  arbitrary. 

3.2.5  Verification:  Verifying  the  integrity  of  the  total  system 
after  implementing  a  computer  program  change  is  the  last  logical  step. 
This  includes  change  to  the  computer  programs  and  documentation.  It  also 
provides  the  point  at  which  the  original  log  or  form  is  revisited  for  the 
last  time  to  close  out  the  corrective  action  process.  Depending  upon  the 
complexity  of  the  change,  additional  test-result  documentation  may  be 
generated  and  stored  with  the  ongoing  software  history.  Other  documenta¬ 
tion  should  also  be  updated  to  change  history  and  function  descriptions  and 
source  listings. 

3.3  Documentation:  Methods  of  documenting  NDCR  to  support  efficient 
computer  resource  modification  and  problem  diagnostic  efforts. 

3.3.1  General:  NDCR  documentation  is  very  important  but  not 
easy  to  categorize  to  everyone's  liking.  In  general,  two  document  types 
should  be  considered:  (1)  a  combined  systems  requirements/design  document 
and  (2)  a  user/operator  guide.  Regardless  of  the  source  of  the  NOCR 
documentation  (i  .e.,  in-house  or  vendor  supplied)  sufficient  documentation 
should  be  developed/acquired  to  efficiently  operate  and  maintain  the 
system. 


3.3.2  System  Descriptions:  Adequately  detailed  documentation  is 
needed  to  perform  efficient  maintenance  on  the  system,  unleTS  the  supplying 
vendor  retains  such  information.  Sufficient  algorithm/Drr'jess  descriptions 
and  source  code  listings  are  essential  for  successful  and  timely  problem 
resolution.  It  is  not  uncommon  for  short  cuts  to  be  taken  in  documenting  a 
system  (e.g.,  hardware,  software  and  interfaces).  Typically  such  docu¬ 
mentation  is  very  “top-level."  However,  the  systems  analyst,  component 
engineer  or  system  engineer  may  not  appreciate  such  brevity  when  a  major 
problem  has  to  be  solved.  Formal  documentation  is  generally  not  needed;  a 
set  of  file  foldeu/notebooks  and  source  cpde  listings  may  be  sufficient. 
But  they  do  need  to  be  well  organized .^ontai n  minimum  information1^? 
accessible.  Using  a  “unit  development  folder"  concept  might  be  the  best 
methodology  for  the  NDCR  management  to  consider. 
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3.3.3  Unit  Development  Folders:  Many  computer  software  devel¬ 
opers  use  a  documentation  and  development  management  tool,  commonly  re¬ 
ferred  to  as  a  unit  development  folder  (UOF)  or  a  programmer's  notebook. 
This  tool  is  nothing  more  than  an  documentation  organization  scheme  de¬ 
signed  to  be  used  In  parallel  to  the  development  of  software.  The  UOF  is 
intended  to  be  used  at  a  low  level  in  the  software  design,  e.g.,  module 
level.  Hence,  a  set  of  UOFs  will  define  a  total  software  design  and 
configuration  status.  The  UOF  is  typically  organized  to  contain  the 
following: 


a.  Introduction.  The  developer  can  place  overall  system 
description  information,  review  and  audit  sign  off  sheets,  overall  software 
sizing  data,  (i  .e.,  how  much  memory  will  be  used  in  program  execution),  and 
development  schedules. 


b.  Requirements.  The  developer  should  extract  each 
requirement  from  system  or  subsystem  specifications  that  pertain  to  the 
particular  module.  In  addition  to  performance  requi rements,  other  impor¬ 
tant  interface  (i.e.,  hardware  or  software)  and  design  convention  informa¬ 
tion  should  be  located  in  this  section.  This  additional  information  serves 
as  a  good  future  reminder  to  the  individual  modifying  a  particular  module. 

c.  Detailed  Design.  Each  module  component,  or  unit, 
should  contain  flow'  diagrams  or  program  design  language  listings  to  show 
the  structure  details.  If  a  large  number  of  units  are  encountered,  it  is 
suggested  to  have  one  preceeding  section  that  shows  all  units  in  a  single 
flow  diagram  or  pictorial  overview.  Along  with  each  design,  the  code 
source  listings  (including  all  comments)  should  be  included. 


d.  Test.  This  section  should  attempt  to  document  the 
test  criteria  and  results  associated  with  each  software  unit  or  grouping  of 
units.  A  thorough  record  of  testing  at  the  lowest  possible  level  becomes 
an  asset  in  future  trouble  shooting  or  module  redesign. 


e.  Change  Record.  A  history  on  the  changes  made  to  the 
module  Is  very  important.  Change  records  can  be  used  to  recover  from  an 
unauthorized  change  to  the  module  configuration.  It  can  also  assist  in 
understanding,  “Why  did  it  work  before  and  not  now?"  type  of  problem. 


j.3.4  Users  Guide:  If  the  development  of  a  support  computer 
system  is  not  tasking  enough,  the  NOCR  manager  generally  is  asked  to  also 
support  the  systems'  use.  The  use  of  a  computer  system  will  be  driven  by 
the  man-  machine  considerations  not  addressed  during  the  design  phase.  The 
user  will  generally  need  to  have  a  broad  pictur-  ,f  how  the  system  operates 
and  adequate  instructions  and  information  to  know  what  is  going  on  and  how 
to  recognize  incorrect  operations.  Once  recognized  as  an  incorrect  opera- 
-ion,  the  user  must  then  know  what  to  do  next.  The  users  guide  should  lead 
the  user  to  some  logical  conclusion  if  nothing  more  than  calling  some  phone 
number  to  report  or  discuss  the  problem. 


3,4  System  Support:  A  methodology  established  to  assure  minimum 

interruption  of  an  automated  systems  usage. 
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3.4.1  Organic  Support:  The  NDCR  managers  needs  to  be  concerned 
about  how  a  particular  computer  system  will  be  supported  and  assure  that 
defined  support  is  properly  rendered.  A  link  should  exist  between  the 
supporting  and  the  developing  activities;  however,  don't  depend  on  it. 
This  is  the  reason  for  good  documentation  and  good  design  practices. 
Having  a  good  staff  of  experts  that  can  agressively  attack  hardware,  soft¬ 
ware  and/or  operator  problems  is  essential .  Likewise,  having  spare  equip¬ 
ment  and  spare  or  backup  experts  can  be  a  significant  factor  depending  on 
the  computer  system  complexity.  However,  in  order  to  work  problems,  the 
NDCR  manager  must  be  assured  to  receive  the  problems,  documented  well 
enough  to  have  a  fighting  chance  to  solve  the  reported  problem.  Hence,  a 
workable  communication  system  should  be  established  using  telephone  "hot 
lines,"  electronic  mail  boxes,  deficiency  or  problem  report  forms,  etc.  to 
capture  user  problems.  The  users  guide  and  a  problem  communication  system 
will  determine  the  user  acceptance  of  the  automated  system  and,  ultimately, 
the  productivity  realized  by  the  organization. 

3.4.2  Vendor  Support:  The  NDCR  manager  should  consider  assign¬ 
ing  one  or  several  employees  to  act  as  liaisons  with  the  vendor  or  vendors 
that  have  support  agreements  with  the  organization.  In  this  manner, 
specific  attention  can  be  applied  to  the  hardware  and  software  problem  re¬ 
solution,  and  the  vendor  performance  can  be  accurately  followed.  Computer 
systems  are  supposed  to  save  time;  if  a  vendor  can't  support  you  so  you 
can't  realize  those  time  savings,  then  a  change  might  be  warranted. 

3.4.3  System  Change:  A  computer  system's  hardware  or  software 
configuration  will  always  change.  Change  begins  before  formal  usage  begins 
and  continues  as  long  as  management  permits  it.  Change  is  generally  good. 

It  indicates  that  an  interest  in  the  system  exists  and  new  ideas  are  not 
being  quashed.  Change  could  also  be  an  indicator  of  how  poor  the  system 
was  designed  or  that  it  has  outgrown  its  usefulness.  The  NOCR  manager 
needs  to  be  very  sensitive  to  the  nature  and  trends  to  suggested  change.  A 
proper  perspective  on  proposed  changes  can  only  be  achieved  if  these 
changes  are  evaluated  with  the  NDCR  and  deliverable  systems  properly  taken 
into  account.  The  totality  of  changes  and  support  problems  may  suggest 
radical  change  vice  a  patch-quilt  approach.  Emotions  will  easily  cloud  the 
real  problems  and  workable  solutions.  This  suggests  the  need  for  a  sepa¬ 
rate  forum  to  handle  the  significant  problems  separate  from  the  daily 
support  activity. 

3.4.4  Responsibility  Assignment:  The  NDCR  manager  should  not 
overlook  explicit  designation  of  responsibilities  to  respective  individuals 
who  directly  affect  the  NDCR.  Assignment  should  specifically  address  the 
integrated  management  of  NDCR  computer  programs,  equipment,  documentation, 
using  personnel  training,  supply  support  arj  support  services.  Never  under 
estimate  the  potential  mismanagement  of  computer  resources;  it  is  too  easy 
to  do.  Of  particular  note,  the  NOCR  manager  should  give  special  considera¬ 
tion  to  the  individual (s)  responsible  for  the  computer  programs . 

Maintaining  a  good,  firm  management  handle  on  the  computer 
programs,  and  associated  documentation,  is  essential.  Computer  programs 
represent  a  growing  investment;  a  valuable  resource  that,  by  the  simple 
absence  of  visible  properties,  needs  protection  throughout  the  life-cycle. 
Assignment  of  responsibility  is  essential.  Assignment  of  resonsibi lity  to 
protect  the  resource  is  only  part  of  the  objective;  establishing  and  main- 
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taining  communications  with  the  other  responsible  resource  managers  is  also 
important.  Particular  areas  of  interest  should  include:  matters  of 
security,  configuration  control,  requirements  analysis,  fault  analysis,  and 
corrective  action,  customer  or  user  support,  special  access  control,  docu¬ 
mentation,  maintenance  and  data  base  design  and  maintenance. 

3.5  Library  Control:  Procedures  to  administer  computer  programs, 

data  bases  and  associated  documentation  to  assure  disciplined  use  and 
maintenance  of  computer  software  resources. 

3.5.1  Procedure  Development:  Library  control  procedures  are 

essential  to  the  integrity  of  computer  programs  and  data  bases.  Whether  a 
library  consists  of  a  few  dozen  cassette  tapes  or  a  building  full  of  mag¬ 
netic  disks,  procedures  are  needed.  Making  the  physical  magnetic  storage 
devices,  labelling  the  programs  or  data  bases  with  control /identification 
numbers  and  dates,  maintaining  physically  remote  storage  of  duplicates, 
running  comparisons  of  differing  programs,  etc.  represent  some  of  the  areas 
where  library  control  procedures  play  a  very  important  part.  Procedures 
should  receive  rigorous  attention  commensurate  with  the  “high  value"  nature 
of  a  library.  For  instance,  computer  aided  design  (CAD),  manufacturing 
(CAM),  inspection  (CAI)  and  related  data  bases  should  receive  optimum 
treatment.  Protection  from  inadvertent  mistakes  or  intentional  alterations 
is  another  significant  factor  in  library  control  procedures.  The  library 
discipline  is  an  insurance  policy  and  one  that  needs  adherence. 

3.5.2  Library  Operations:  In  preparing  library  procedures,  the 
NOCR  manager  should  consider  addressing  several  specific  tasks  or  pro¬ 
visions.  First,  consider  the  mechanics  for  the  library  function.  As  a 
service  activity,  the  library  must  be  responsive  to  the  user's  needs  which 
includes  a  measure  of  discipline.  Second,  assure  that  the  library  data  is 
backed  up  and  that  the  backed  up  information  is  physically  stored  remote  to 
the  library.  Fire,  sprinkler  systems,  electromagnetic  distrubances ,  tor¬ 
nados,  and  floods  are  all  potential  catastrophies  that  can  rendor  a  good 
library  useless.  Third,  keep  documentation  hand-in-hand  with  computer 
programs.  By  segregating  the  two  or  creating  two  separate  systems  will 
only  enhance  the  potential  for  miscommunication .  Fourth,  think  security. 
Unauthorized  change  to  computer  systems  and  its  software  is  not  too  diffi¬ 
cult.  Even  easier  is  the  ability  to  copy  computer  programs  which  might 
violate  copyright  laws  or  present  a  threat  to  trade  secrets. 

3.6  Subcontracting:  Establishment  of  a  methodology  to  assure  that 
subcontracting  practices  aequately  address  NOCR  management  discipline. 

3.6.1  The  Subcontracts:  Should  a  manager  be  concerned  with  NOCR 
when  subcontracting  for  a  product-  or  service?  In  today's  automation  en¬ 
vironment,  the  answer  is  probably  yes!  Oo  you  have  a  warrantee?  If  so, 
how  long?  Or,  you  may  have  a  support  agreement,  but  should  you  be  paying 
for  it  in  the  first  place?  NOCR  can  probaly  be  found  behind  most  products 
procured  today.  The  bottom  line  question  might  be,  "How  concerned  is  the 
subcontractor  about  managing  NDCR? "  In  contracting  for  a  product  or 
service,  it  might  pay  to  study  the  subcontractor ' s  behavior  regarding  NOCR. 
This  might  be  the  time  to  address  specific  NOCR  issues  either  in  the 
contract  or  as  a  business-to-business  agreement.  When  using  the  contract 
vehicle,  get  relevent  NOCR  information  as  soon  as  possible,  most  preferably 
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in  a  planning  document. 

3.6.2  Managing  the  Subcontract:  Understand  the  role  automated 
systems  will  play  in  the  development  and  delivery  of  a  product.  A  sub¬ 
contractor's  automated  systems  are  most  likely  the  reason  you  contracted 
with  them  in  the  first  place.  The  subcontractors  management  attention  to 
those  automated  systems  represent  an  indicator  of  NDCR  management  health. 
The  contract  itself  will  inform  the  customer,  to  a  limited  degree,  of  the 
NDCRto  be  used  by  the  subcontractor,  providing  NDCR  was  addressed  in  the 
first  place.  The  rest  of  the  information  will  have  to  come  from  customer  - 
subcontractor  discussions  and  face-to-face  visits.  Each  stage  of  a  product 
procurement  necessitates  an  understanding  of  the  NDCR  involved  and  the 
potential  risks  at  hand,  not  just  the  productivity  enhancements  advertised 
by  company  public  relations.  Dig  into  how  changes  are  controlled  for  a 
computer  aided  manufacturing  system.  Understand  the  particular  specifi¬ 
cations  or  requirements  set  into  a  computer  aided  process  controller  and 
what  protections  exist  to  prevent  unauthorized  changes  to  the  process. 
Find  out  how  acceptance  criteria  will  be  programmed  into  an  automated  test 
system  and  determine  how  and  what  is  the  nature  of  raw  data  outputs. 
Examine  the  role  of  methodology  in  setting  standards  and  evaluating  cali¬ 
bration  or  correction  factor  data  files  used  in  testing  systems.  Question 
the  credibility  of  simulation  and  analysis  programs  used  with  a  computer 
aided  design  (CAD)  system  and  find  out  how  the  CAD  data  base  is  handled  to 
prevent  unauthorized  changes. 

4.0  SYSTEM  CONSIDERATIONS:  There  are  various  applications  of  computer 
resources  (CR)  that  typically  fall  into  the  category  of  NOCR .  With  any 
specific  CR  application  will  come  some  special  management  considerations. 
For  the  most  part,  the  NDCR  manager  will  be  exposed  to  the  level  of  disci¬ 
pline  that  needs  to  be  exercised  for  an  application.  With  management 
discipline  comes  the  hazzard  of  over  constraint,  a  concern  that  has  a  good 
basis  for  management  attention.  Likewise  will  come  the  different  support 
considerations  both  for  the  equipment  and  software,  and  the  users.  Final¬ 
ly,  a  manager  will  need  the  visibility  of  the  resources  at  hand  and  will 
need  various  tools  to  assess  the  health  of  the  management  commitment. 

4.1  Automated  Test  Systems:  Automated  test  systems  (ATSs)  should 
receive  rigorous  management  treatment  to  assure  the  integrity  of  ATS 
hardware,  software,  and  associated  documentation  and  ATS  interfaces. 

4.1.1  ATS  Impact:  ATS  plays  a  major  role  in  the  acceptance  of 
daliverable  products  at  all  levels  of  build-up  and  test.  In  some  cases, an 
ATS  may  be  as  complex  and  essential  as  the  deliverable  product  being 
produced  and  tested.  Management  should  ensure  that  a  rigorous  life-cycle 
treatment  is  applied  to  ATS  again  commensurate  with  the  project  risks. 

4.1.2  ATS  Plans:  ATS  development  is  an  integral  element  to 
overall  program  planning.  Such  development  should  be  defined  by  a  develop¬ 
ment/product  specification,  hardware  and  software  configuration  documents 
that  are  adequately  defined  and  controlled.  Also,  the  ATS  should  be 
rigorously  val i dated /veri fied  for  all  test  requirements,  demonstrate  its 
accuracy  and  precision,  and  be  highly  maintainable.  Management  should  pay 
close  attention  to  an  overall  ATS  usage  strategy,  that  is  look  beyond  the 
initial  ATS  appliction.  With  new  equipment,  sensors  and  advanced  computer 
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availability,  the  ATS  investment  will  not  be  static.  Management  needs  to 
look  at  long  range  transition  strategies  to  capture  basic  hardware,  soft¬ 
ware  and  documentation  designs.  Reusable  software  is  a  potentially  signif¬ 
icant  cost  saver.  Employing  a  modularized  ATS  configuration  employing 
standard  data  busses  and  hardware  connection  could  be  significant.  Commit¬ 
ting  to  a  high  order  language  as  PASCAL,  ATLAS,  Ada,  etc.  might  also  be 
significant  in  reducing  computer  program  development  costs. 

4.1.3  ATS  Controls:  ATS  usage  should  be  properly  controlled 
through  automated/manual  procedure,  adequately  maintained  and  calibrated, 
adequately  protected  to  avoid  unknown  hardware /software  configurations,  and 
properly  documented  testing  conditions  to  assure  test  repeatability. 

4.1.4  ATS  Integrity:  ATS  computer  programs  should  undergo 
rigorous  library  controls  and  corrective  action  discipline.  Built-in  test 
capabilities,  calibration  routines,  problem  diagnosis  programs  and  ATS 
self-test  results  should  receive  particular  control  emphasis  to  assure  ATS 
integrity. 


4.1.5  ATS  Documentation:  ATS  user  and  design  documentation, 
including  ATS  training  materials,  should  also  be  properly  developed  and 
controlled.  Efficient  use  and  effective  maintenance  of  the  ATS  relies 
heavily  on  this  factor.  Reluctance  to  invest  indocumentation  is  the  most 
serious  hazzard  facing  managment.  Many  times,  an  ATS  may  have  a  specific 
application  for  only  one  to  two  years.  After  this  period,  the  ATS  may 
scrapped  or  completely  reconfigured  for  another  project.  But  be  observant. 
One  might  sell  the  hardware,  erase  the  software  media,  but  the  documented 
algorithms  and  basic  ATS  configuration  designs  (both  hardware  and  software) 
will  most  probably  be  used  again.  In  fact,  long  term  NDCR  management 
should  have  addressed  "software  reusability"  at  the  start  to  reduce  rein¬ 
venting  ATS  designs,  implementation  and  usage  instructions. 

4 .2  Computer  Aided  Design  Systems:  A  computer  aided  design  (CAD) 
system  should  receive  adequate  management  attention  to  maintain  the  integ¬ 
rity  of  CAD  analysis  computer  programs,  user  instructions,  operating 
systems  and  generated  data  bases. 

4.2.1  Data  Bases:  The  generated  CAD  data  base  that  describes 
the  design  of  a  product  warrants  very  stringent  controls.  NDCR  managers 
should  establish  very  explicit  procedures  in  establishing,  changing  and 
certifying  a  CAD  data  base.  The  media  does  offer  additional  risks  in 
maintaining  the  purity  of  a  data  base  and  confidence  in  the  controls 
established  by  management.  The  first  risk  is  in  the  acceptance  of  design 
data  into  a  controlled  data  base.  Is  the  design  ready  for  rigorous  con¬ 
trol,  or  is  it  long  overdue?  Management  should  consider  requiring  that 
various  checks  or  vteps  have  been  completed  before  accepting  a  design  for 
control.  For  example,  have  certain  analyses  been  done  and  documented? 
Have  certain  interfaces  (e.g.,  jigs,  tools,  software)  been  examined  and 
results  documented?  Second,  once  the  data  base  has  been  accepted,  who  is 
authorized  to  make  changes  and  what  procedures  are  to  be  followed?  Are 
back-up  copies  of  the  data  base  kept  at  a  remote  site?  Are  old  versions 
retained?  When  are  updates  performed  and  how  often  permitted  or  required? 
Finally,  who  can  have  access  to  the  data  base,  even  if  only  on  a  read  only 
basis?  Proprietary  or  security  constraints  should  be  of  vital  concern. 
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4*3  Computer  Aided  Manufacturing  Systems:  A  computer  aided  manu¬ 

facturing  (CAM )  system  should  recei ve  adequate  management  attention  to 
maintain  the  integrity  of  CAM  computer  program  support  tools,  operating 
systems,  user  instructions,  safety  features  and  used  data  bases. 

4.3.1  CAM  Data*  A  set  of  commands  designed  to  drive  a  particu¬ 
lar  numerically  controlled  (NC)  device  poses  a  configuration  management 
challenge.  Once  a  file  has  been  established,  whether  by  manual  or  auto¬ 
matic  means,  an  operator  is  faced  with  the  problem  of  retrieval,  identifi¬ 
cation  and  verification.  Finding  a  particular  data  file  might  be  very 
difficult  unless  stored  on  a  separate  media,  i.e.,  paper  tape,  diskette, 
etc.  Even  if  the  file  is  accessible,  the  next  question  is  whether  the 
content  of  the  file  reflects  the  correct  version.  Finally,  feedback  is 
necessary  in  verifying  that  the  final  product  was  properly,  made  for  a  given 
rC  output  file  used,  i.e.,  a  process  to  avoid  the  "impossible"  errors. 
Herein  rests  the  challenge;  each  one  of  these  steps  will  involve  man- 
machine  interaction.  The  opportunity  for  “garbage-in,  garbage-out"  will 
exist.  The  NC  machines  are  not  intelligent  (yet)  to  catch  obvious  input, 
set  up  or  command  errors.  One  must  pay  particular  attention  to  each 
man-machine  step  and  build-in  preprocessing  checks  to  detect  common  errors. 
Independent,  data  entry  processes  in  addition  to  simulations  might  be 
considered  as  a  trade-off  to  numerous  "dummy"  product  manufacturing  dry 
runs.  Checking  of  "to-be-used"  files  with  history  files  for  command-by- 
conmand  accuracy  might  be  implemented  when  high-value  materials  or  extreme 
precisions  are  required.  If  one  is  using  a  distributive  system,  it  might 
be  useful  to  use  fault  detection  routines  at  NC  site  to  assure  accurate 
reception  of  data.  (Remember  the  CAM  environment  is  typically  very  noisy 
and  can  have  drastic  effects  on  simple  communication  methods.) 

4.3.2  CAM  Safety:  Safety  is  a  must.  Computer  controlled 
processes  provide  a  greater  opportunity  to  link  many  previously  separate 
processes  into  one  continuous  set  of  operations.  If  the  human  must  inter¬ 
act  with  the  NC  process,  care  must  be  taken.  Typical  integrations  of  robot 
devices  wtih  "traditional"  NC  devices  also  should  enhance  the  managers 
safety  concerns.  But  what  to  do  about  it7  Perhaps,  the  best  process 
should  not  involve  human-machine  interaction  during  any  sequence;  such 
interaction  should  occur  only  after  the  computer  has  released  control  of 
the  machine.  As  an  alternative,  one  should  consider  software  routines  to 
evaluate  any  given  NC  process  to,  if  nothing  else,  identify  potentially 
hazardous  operations. 

4.4  Computer  Aided  Inspection  Systems:  A  computer  aided  inspection 
(CAI)  system  should  receive  adequate  management  attention  to  maintain  the 
integrity  of  CAI  computer  programs,  operating  systems,  user  instructions, 
and  used/devil  oped  data  bases. 

4.4.1  Criterion:  In  any  inspection  activity,  an  actual  measure¬ 
ment  will  be  compared  to  some  expected  measurement  with  an  associated 
tolerance.  Established  or  expected  measures  form  a  criterion  data  base 
whose  accuracy  is  a  must.  With  any  computer  comes  the  configuration 
management  problem.  Do  I  have  the  right  version?  Has  anyone  unauthorized 
altered  my  data  base?  Is  the  criterion  being  applied  to  the  proper  coordi¬ 
nates  or  locations  in  or  on  the  product?  These  are  typical  configuration 
concerns  that  are  extremely  important  to  address  where  a  CAI  system  is 
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involved.  Rigorous  procedures  in  establishing,  entering  and  maintaining 
criterion  must  be  on  hand.  If  possible,  build  such  safeguards  into  the 
computer  programs  used  to  interface  with  the  human. 

5.0  MEASUREMENT  PRACTICES:  Just  because  the  NOCR  manager  is  concerned 
about  such  resources,  are  the  resulting  practices  adequate  to  satisfy 
management’s  objectives?  Perhaps  the  established  management  practices  are 
too  constraining  or  are  ill-defined.  What  if  changes  to  the  practices  are 
implemented,  will  the  results  on  the  personnel  and  computer  resources,  both 
good  an  bad,  be  visible?  Management  will  need  some  form  of  measurement  to 
address  these  questions.  Common  measurement  methods  involve  evaluation, 
inspection  and  auditing;  terms  that  can  strike  terror  into  the  heart  of  a 
manager .  So,  how  might  one  address  needed  measures  on  NOCR  management  and 
not  cause  cardiac  arrest?  Perhaps,  the  following  guidelines  can  help. 

5-1  Measurement  Organization:  The  process  of  measurment  can  general¬ 
ly  be  satisfied  through  normal  supervision  activities  or  by  establishing  a 
separate  organization/activity.  Consider  methods  where  "independency" 
exist,  whether  done  by  a  computer  (i.e.,  automatic  measurement)  or  by  an 
evaluator .  Independency  helps  to  reduce  significant  skewing  due  to  per¬ 
sonal  or  "directed"  biases.  Even  if  independent  measures  are  achieved,  be 
very  careful  of  the  handling  of  the  acquired  information.  Constructive 
uses  of  the  measures  is  absolute;  primitive  uses  only  destroy.  Management 
should  consider  a  feedback  loop  for  measurement  information  to  the  lowest 
possible  level  of  an  activity  and  not  just  upward  flow  or  "black -hole" 
collection  of  information. 

Acquiring  the  information  should  be  as  integrated  into  ones  way 
of  doing  business  as  possible.  Collecting  specific  information  at  the 
computer  level  and  as  automatic  as  possible  is  the  ideal  situation.  Using 
the  computer  to  assist  in  colection,  processing  and  feedback  is  generally 
desirable.  Just  be  careful  of  sensitive,  classified  or  proprietary  in¬ 
formation.  Measures  should  reflect  various  system  characteri sties : 
configuration  status  of  hardware,  software  and  documentation;  personnel 
producti vity/performance  and  compliance  with  established  rules;  and  product 
conformance  (objective  and  subjective).  Try  to  keep  a  reasonable  perspec¬ 
tive;  be  serious  with  your  management  objectives.  To  measure  NOCR  manage¬ 
ment  system  characteristics  effectively  requires  "tailoring."  What  bounds 
tailoring  will  be  rather  subjective.  There  is  no  magic  formula.  The  best 
guidelines  are  keep  it  simple,  make  it  work  for  you,  and  get  a  good  cross 
section  of  all  system  characteristics  . 

5*2  Evaluating  Planning:  As  NOCR  use  increases  within  a  development 
environment,  management  should  evaluate  how  well  the  NOCR  was  integrated 
intr  the  overall  deliverable  product  planning.  Certainly,  reviewing  past 
- — gram  problems  with  NOCR  or  its  program  integration  are  necessary  to 
preclude  similar  future  problems.  However,  as  with  any  new  applied  tech¬ 
nology,  developers  should  evaluate  how  the  potential  risks  to  the  program 
have  been  ^addressed.  Automation  advances,  for  the  most  part,  create  a 
potential  for  significant  cost/schedule  impacts  resulting  from  inadver¬ 
tent  or  intentional  alterations  to  computer  programs  and  data  bases. 

5.3  Audit! ng  Correct i ve  Acti ons :  Writing  procedures  to  correct 
problems  to  NUCRs  is  only  the  first  step  for  management.  Making  sure  that 
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the  discipline  has  been  adhered  to  is  essential.  Many  NDCRs  comprise 
systems  that  service  other  company  employees.  User  problems  necessitate  a 
responsive  and  thorough  corrective  action  system.  A  disciplined  approach 
to  corrective  action  should  be  maintained  to  avoid  causing  new  problems 
while  fixing  old  ones.  Likewise,  the  decision  to  implement  a  change  or  not 
should  be  an  integral  part  of  the  corrective  action  system  either  through 
the  control  of  a  responsible  manager  or  a  group  of  key  players.  Verify 
these  activities  are  properly  supporting  the  users  is  the  bottom  line 
objective. 

5 .A  Evaluating  Documentation:  Feedback  from  the  user  is  generally 
the  best  measure  or  how  ef f ecti ve  computer  resource  documentation  really 
is.  Getting  feedback  is  the  first  challenge.  Understanding  and  using  the 
feedback  is  the  second  challenge.  Characteristics  of  NOCR  documentation 
that  should  be  evaluated  might  include  accessibility,  readability,  algo¬ 
rithm  descriptions  (degree  of  depth),  general  logic  flow,  test/validation 
information,  requi rements  information  and  interface  information. 

5.5  Auditing  Management:  Being  self-critical  is  never  easy,  espe¬ 
cially  for  management  when  everything  seems  to  be  going  well.  Automation 
can  lure  management  into  a  false  sense  of  security.  What  might  be  a  small 
problem  frequently  multiplies  into  one  or  a  number  of  big  problems  when  not 
treated  in  a  disciplined  fashion.  Certainly,  watching  how  management  copes 
with  such  small  problems  can  be  amusing.  When  the  problems  are  no  longer 
small  and  costs  mount,  the  humor  can  diminish  rapidly.  Checking  manage¬ 
ment's  resolve  to  follow  its  own  prescription,  directly  reflects  the  health 
of  the  business.  Perhaps  it.  is  only  common  sense,  but  it  is  only  as  common 
as  management  allows  it  to  be. 

5.6  Auditing  Library  Controls:  Auditing  library  controls  is  a  safety 
check  or  an  insurance  check.  NtiCR  computer  programs  represent  a  sizeable 
monetary  investment  and,  frequently  are  critical  to  the  effectiveness  of  an 
activity.  Again,  checking  one's  own  procedures,  whether  simple  or  complex, 
reveals  the  health  of  the  control  discipline. 

5.7  Measuring  ATS:  The  automated  test  system  (ATS)  is  a  very  impor¬ 
tant  component  to  the  product  delivery  process.  All  the  checks  and  bal¬ 
ances  placed  on  getting  an  ATS  working  and  keeping  it  going  should  be 
complemented  with  an  oversight  activity.  Again,  compliance  with  estab¬ 
lished  procedures  and  identification  of  "bad"  practices  are  the  primary 
objectives  for  measuring  related  ATS  procedures.  Common  sense  dictates 
rigor,  providing  the  product  developer  is  interested  in  producing  a  qual¬ 
ity,  deliverable  product.  Characteristics  that  might  be  addressed  during 
procedure  audits  and  reviews  are: 

a.  Adequacy  of  ATS  documentation  that  is  needed  to  interface 
with  the  product  to  be  tested  (e.g.,  interface  equipment  and  calibration 
factors.) 

b.  Adequacy  of  the  test  input/output  documentation  (e.g., 
supportive  of  test  repeatability  and  diagnosis  of  problems). 

c.  Adequacy  of  procedures  to  determine  the  precision  and  ac¬ 
curacy  of  the  ATS. 


12 


d.  Adequacy  of  ATS  library  controls  and  ATS  corrective  action 
activities.  Also,  how  do  ATS  problems  interface  with  tested-product 
problems? 

5.8  Evaluating  Subcontract  Management:  Any  customer  wants  to  be 
assured  of  a  good  product /service  from  a  subcontractor.  The  supporting 
cost  (i .e.,  in-house,  nondeliverable  resources)  might  not  get  the  proper 
attention  unless  addressed  by  the  item  developer.  The  quality  and  sub¬ 
contractor  interest  in  the  NOCR  will  be  a  significant  indicator  of  the 
development  process  health.  In  addition,  the  dependence  upon  NDCR  by  the 
deliverable  product  for  maintenance  may  be  an  off  stage  factor  and  waiting 
in  the  wings  not  to  be  revealed  until  after  product  delivery.  The  un¬ 
pleasantness  associated  with  product  deficiencies  tied  to  NDCRs  or  the 
revelation  of  "unknown"  backup  resources  are  always  a  source  for  heated 
conversations  and  wasted  revenue. 

6.0  MANAGEMENT  BENEFITS:  Conmitment  to  managing  NOCR  extends  the  basic 
principles  of  planning,  responsibility,  and  control  deemed  applicable  for 
deliverable  resources.  It  instills  the  fact  that  NOCR  are  not  mechanical 
devices,  but  complex  electromechanical  systems.  This  raises  important 
management  concerns  about  risk  in  the  use  of  NDCR  to  the  deliverable 
cost/schedule  factors.  It  also  associates  the  same  "life-cycle"  concerns 
identified  for  deliverable  weapons  systems  and  urges  maintainability  to  be 
a  part  of  the  NDCR  development  consideration.  As  is  fundamental  to  many 
computer  systems,  protection  of  the  computer  program  and  data  information 
(both  written  and  computer  stored)  becomes  a  focus  for  efficient  use  of 
NDCR.  All  these  factors  point  to  the  protection  of  the  productivity 
enhancement  gained  by  automating  a  process.  Hence,  a  substantial  intang¬ 
ible  cost  saving  is  realized  through  preventative  measures.  Also,  direct 
cost  savings  are  achieved  through  the  implementation  of  a  standard  manage¬ 
ment  approach  within  an  activity.  Employee  transfer  with  the  activity  will 
require  less  reorientation/training.  To  gain  efficiency  within  an  organi¬ 
zation,  management  can  centralize/decentralize  such  functions  as  software 
libraries,  configuration  control,  procedure  auditing,  corrective  action 
control /monitoring,  documentation  support,  and  subcontractor  evaluation. 
Although  a  cost  is  incurred  with  implementing  and  maintaining  such  func¬ 
tions,  such  costs  are  substantially  less  than  doing  activities  without 
automation.  And  incurring  such  costs  are  necessary  if  the  automation 
benefits  are  to  be  sustained  for  any  period  of  time. 

7.0  MANAGEMENT  CONCERNS:  With  the  adoption  of  any  discipline  comes  a  cost 
of  its  implementation  and  maintenance.  Implementation  costs  will  be  a 
direct  reflection  of  the  formality  management  elects  to  adopt.  Implementa¬ 
tion  costs  will  normally  be  one-time.  Maintenance  costs  will,  again,  be 
incurred  and  reflect  the  formality  adopted  by  management.  Hence,  with  some 
added  costs  expected,  management  could  inflate  these  costs,  whether  de¬ 
liberate  or  not,  through  the  misapplication  of  the  management  objectives. 
For  example,  management  could  over  control  activities,  preparing  excessive 
procedures  or  developing  restrictive  policies  in  a  business  environment 
where  simple  policies/procedures  could  suffice. 
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% 

AA  ACQUISITION  QUALITY  ASSURANCE  (OHentation )  - 
Faml 1 larity  with: 

usa.  001.  Oefense  Acquisition  Regulation  (OAR)  purpose.  |tructure  and 

and  specuf * types^nd^methods  t^nd ' se ife  tioITof 'contractors  ? *  "e90tUt,°" 

and  contract  structure  ""includi  no  ^format ' ' xed  prIc-*  cost  reimbursement); 
precedence,  changes !  ciauses!  eu.  ’  number,n9.  «>*<■.  order  of 

stem  l^fe'cycle/establfshment^^Program^roject  Offices'  a*?56  °f  f6 

ogram  management  responsibilities.  9  oject  0ffices»  and  general 

policies ^re  1  a  tecT  to^ass  i  gnment  °of  ^on  t  ^  f r^H s  ?  rT 1  c 2S ,  A i  r  Force  and  AFSC 
contract  administration  9  '  ract  a<,m,ni5tration  and  supporting 

""*»  sPPcified'by  DAReand°theirtrelated  W 

WL?sfo“l3S 

MU  5-. 2/79,  AFSC  unique  requirements  and  NASA  documents.  ’ 

standards^and^poUctes  % .W^StST^?  SP"if 

010.  Program  Office  control  of  contracts:  post-award  meeting* 
contract  administration  delegations/MOAs/letters  of  instruction*  program 
management  reviews  and  engineering  reviews.  progra,n 

Oil.  In-plant  Contract  Admin istration  QA  proqram.  i  e  DAR 

i|V’  Policies  and  procedures  (including  AFCMDRs  74-1 

1/8-1).  Development  of  QA  plan  for  contract  implementation. 


* 


Jl'ifc  *i5  snd^Proceduresf ° Including0^ SCR*" 74-1  j.  W/^SC  ‘l'“  *fsi  Comi’'1""1 

„-.i,,M°13;JUa!,!yJSUr!e?S,b^  contractors ,  Government  CAS  and  buylnq 
s’  and  rslated  scheduling,  interface  coordination  and  conduct.9 

. •  ,^i4,  Configuration  Management  policies  and  procedures  specifica- 

?i;Cr«ToS“A^fSS^Civi,!oe,“*tloB  and  chaT  contr»'- ;««- 

015.  User  Quality  program  and  feedback  systems . 

;3  qc.’IcSAL  COMMUNICATION  SKU I  S 

^i:J:,on0J;oc«"!,UrUy  WUh  th'  ^chologlcal  aspects  of  the  com- 

efi^Ue’S^S 

the  desired  reader  reaction  In^acUo^  'nfon"ation  "««sary  to  assure 

Abnity  t0  orally:  convey  information  to  achieve  the 

o  ;  f  c-o  4eeSn  ^T,y  and  c™cise,y  t0  questions,  and  cope  with 
ooc.i  fv.ce-t.o-.  a«.e  and  telephone  conmunication  problems. 

ability  to°°b:stende7fecUvely?f  C°“0n  ,iSteni"9  Jnd,  Tarr  fau,ts'  *"« 

Hte-personai  ne^If SU"S  ^ 

-c  .-UMA.N  RELATIONS 

ss^JSpi’ss  «asw?K3r« 

-tthers,  anS^ee^i^s  f o°r ' imp^e^nt  " 1“' *££4.'*"  !>ehavior  ,n 

--rn.-i-a?ion  srArai^L!°-r2CT,ZS  n,e,ls  of  others'  and  employ  appropriate 
:Iw  ’l7  ,  eg,eS  in  order  t0  obtain  willing  cooperating  con- 

t>  Jer  ..ton  and  respect  of  persons  interacted  with  9’ 


KNOWLEDGES,  SKILLS  4  ABILITIES 
(KSAi ) 


2  NON- TECHNICAL  SPECIALIZED  DISCIPLINES 
-'0  IN-PLANT  QUALITY  ASSURANCE 

001.  Understanding  of  the  in-plant  Acquisition  Dual i tv  Assurance 
Program  as  prescribed  in  OAR,  000,  AF  and  AFSC  directives.  4  CS 

ments  of  AFRJ'74Tl0Tt?»k;a"!edSen0»fi,?nd  t0  comP'^  "'th  require- 

AFSCR  74-2,  AFCNORi  74-1  and  i78-l."C  C0"p0"snt  PeCU,,ar  directives,  i.e. 

AE  CONTRACT  ADMINISTRATION 

**  "  - -  -  _  _  w 

Knowledge  of: 

001.  000  organization  for  Contract  Administration. 

Officers  (,pcOs)Aand°Administrati'vePContactingeofffcersn(ACOs}^°ntraCt^n^ 
faml  1  iar i ty^wi thMiost/ris^shar  ing  trluZ C?'F*  *"d 

Of  precedence,  enamel, ‘  cuSses^^tc?''1”’"’9  CDRL-  «"* 

r1,  ,  °05,  How  contract  modifications  are  authorized  i  e  charms 

agreements?^  *eth°dS  °f  ”°difications'  i.e.,  change  orders  ^’sfppStal 

/  /  0 

pricing  tSS?,  anf  J&ScSl .C°"tr3Ct  S“‘  Pri"C,p,eS  a"d  and 

,  ,  .  f  °07:  icies  and  procedures  for  evaluation,  prior  to  contract 

award  of  contractors'  capabilities  to  meet  the  terms  if  the  «nt“ct 

008*  DAR ,  Air  Force  dnd  AFSC  policies  related  tn  A^innmonf 
contract  administration  and  supporting  contract  administration.9 

losoPMes°and  S^olu^n!^ C  Qua"ty  COntro1  and  Qua,it*  As“ 

and  their°ieiatedVclluIesS  °f  COntraCtual  QA  r^1rements  specified  by  DAR 

i  -<  Inspection,  acceptance  and  warranty  provisions  of  contracts 

including  Government  and  contractors'  rights  and  responsibilities 


J 


vs  fixed  pike  contracts?5  ^  *"d  ,nspsaip"  prov,sio"s  <"  «« 

013,  Other  CAS  functions  related  to  Quality  a  o  •)Jkrrmf -.a*.* 
!*;:  '-Ta0^  and  procurement  systems  approval;  (Lvernmen^prope?t£?  *  *  ’ 

ot OujcJ ^n/, manufacturing  surveillance  and  Priorities  and  Allocations 

*7  5  ^  •  *5  • 

014.  Methods  for  resolution  of  disputes,  claims  and  controversies 

,  015.  Procedures  for  termination  of  contracts  for  default  bv  the 
contract. or  or  convenience  of  the  Government..  J  *■ 


conduct. 


016.  Ethical  considerations  for  CAS  personnel  and  standards  of 


Ar  INSPECTION  SYSTEM/QUALITY  PROGRAM  REquiRFMFMTS 

v i L-Q- 935S?1 ’  Comprehensi ve  knowledge  of  all  provisions  of  MlL-r-45208  and 

n002,  .Understanding  of  Government  Interpretations  of  the  MIL  - 1  - 
-  u  -iw-Q  requirements  as  reflected  in  Military  Handbooks  H-50  and  H-51. 

C03.  Understanding  of  the  language  of  these  SDeci ficatinn'?  in 
tcr-B  of  contractual ly  enforceable  vs  non-Inforceabl?. 

contract  ?m°?;ct?"0“,ed,e  °f  USAF  P°"cy  °f  plac,n3  ^  P'*"=  »"  contract  and 

ton  to  4?Ln requirements?°ntraCt0r  P~ 

OOfi.  Familiarity  with  NASA  publications  NH3  5300.4(13)  and  (1C). 


/ 

/  *  0 


AG  NONCONFORMING  material/corrective  action 

action  Pr^l;,o^ro^e!l?i:!lr^|3|l:,^!<1f58Sl:C^0^ni3^?^JiJ/^Srnctjre 

►  v/oonJfnrCvnPrehrSi -e  knowled9e  of  "Corrective  Action  and  Oisposi- 
ror  'Unconforming  Material”  requirements  of  MIL -STD- 1 520* 

-an  .  •  f?lt‘  t0  evaluate  adequacy  of  contractor  pro-edures  wri*. 

tan  to  implement  the  specifications  and  standards  requirement 

■5f  CAII3RA7I01;  SYSTEM  RE0UIREKFNT5 

001.  Understanding  of  calibration  system  requirements  imposed  by 


cont-acts  (as  specified  in  MIL-I-45208,  MIL-Q-9853,  MIL-STD-45662 
interpretations  of  HD6K-52,  and  NASA  publications)  including: 

'national  standards?®1'1*10"5*1*’  betW*en  cert1f1ed  measurement  standards  and 


b.  Accuracy  of  standards. 

c.  Use  of  calibration  labels  on  measuring  and  test  equipment. 

d.  Calibration  certifications  and  reports . 


1Jli>1(n  _f  e;  Ca  1  ibrat-ion  procedures  for  production  tooling  used  as  a 
i»Jijm  of  acceptance  inspection.  •*  J 


irol^npnt°‘ho  rS1,ty  evaJuate  contractor's  procedures  written  to 
.gl-ment  -.he  contractual  system' requirements ,  and  the  written  calibration 
procedures,  and  the  ability  to  evaluate  for  compliance.  5  calibration 


subcontract  quality 

_  fc  Understanding  of  basic  provisions  of  OAR  relatlna  to 

subcontracting,  ^eluding  make-or-buy  decisions,  consent  Reviews9,  procure- 
y  -Ti  reviews,  contractor  responsibility  for  subcontractors,  etc. 

aool-d  c?ori‘^^nled9i  °f/ar1ouscontractual  provisions  which  may  be 
m?  mn  LcS  ated  t0  C>uall‘ty  of  subcontracted  supplies  e  q 

MIl-  -4o20S,  MIL-Q-9853,  MIL-STD-I535,  and  NASA  Quality  publications! 

003.  Comprehensive  understanding  of  requirements  for  cont-artnr'e 

purchJ 

inspection  of  products,  and  corrective  action.  * 

005.  Ability  to  evaluate  the  adequacy  of  contractor's  documented 
COT?Han?e  theJeto.^  subcontract-^al  1  ty,  its  effectiveness  and  their 


°0b\  Understanding  of  Government  rights  and  responsibilities 
el..:d  to  subcontractors ,  including  standard  inspection  clauses  riaht- 
and  .imitations,  Government  review  of  contractor  purchase  docum^tation 

P-owr'eww %  IVltllc?.  Supp0rti"9  contract  a«")nlstratton  or 'Government 
to  impos Ing^Government 


1u 


QUALITY  SURVEY  CONCEPTS  AND  TECHNIQUES 

,2°;*  Understand  purpose  of  surveys  conducted  by  various  organi- 
(  r0graQ  ?r0Ject  0fnce«  cognizant  AFPRO,  OCAS,  or  other  organizations, 


u^stand  the  essential  elements  of  a  survey  such  as 
..spar  at  ion,  conduct,  reporting  and  follow-up.  '  Cn  * 

acting  survey. Knowledge  of  planning  and  research  required  prior  to  con- 
wvalop  them!  Understandin9  of  "e^  for  checklists,  and  the  ability  to 
5uC1  ,5  iS0sHoi?l1U;  t0  appl?  effective  techniques  In  conduct  of  surveys 

006.  Ability  to  prepare  written  reports  of  survey. 

- I :-rnat Ive^tho^r" for" foHow-u^action^  °bta,",n9  C°rr!“f’e  «tf°"  - 


ngs 


SOFTWAHc  QUALITY  ORIENTATION 

c=^ng>  conficuration^corrtrol^an^veriflcatior^and^a'Mdatlonr*3*'6  de5'9"' 

0.,.,,  a.xiiawssa,?- 

w&arsssi'usrjagr  ~ 

Al  CONFIGURATION  MANAGEMENT 

the  Slslnner!"(Co9nfa?gum??nnS 

witi  Configuration  Status  Accounting  Management 

'  ne'i-nM^r;  WhlCh  Pt0Vldes  for  records  of  actions  accomolishefun^ 
^  1,1cat'cn.  control-  and  audit  functions  of  Conf iguration  Management. 


_ _  -  u°3'  ran,iliarity  with  reviews  and  audits  performed,  includlna 

°eSi9n  Re7ieW3*  FunCtlSnal  *"d  Physical 

005.  Understanding  authorities  and  resoonslbll  1t1as  of  thp  'Fir 
^  ^b-components  in  the  various  aspects  of  Configuration  ' 

PrOjR^y/Pp.ojECT  OFFICE  (Purchasing  Activity) 

cycle°of  "aJ°r  pfiaSSS  °f  •“'»***  <"  th. 

issJt- 

>"-or-1i-S‘w  ^der^and1n9  of  the  principal  functional  processes 

6d  vjUrlng  the  v^ious  phases  of  the  acquisition  cycle,  e.g. 

engineering  Management 
Configuration  Management 
Test  and  Evaluation 

Manufacturing  and  Production  Management 
Integrated  Logistics  Support 
Oata  Management 
Etc. 

:..vw  wr&  %  &  r„^^^tas^ra;ion 

<fi,,'°063  Knowledge  of  Program  Management  Directives  (POM)  Af  SvstPfr* 
(&WC)!"  C0UnC"  (ASARC)-  «•'•»»•  Swt«  Acquisition  & 

007.  Knowledge  of  Statement  of  Work  (SOW)  documents. 

003.  Ability  to  tailor  QA  program  requirements  in  SOWs. 

,  .  „  .  QQ-’  Knowledge  of  contract  clauses,  Source  Selection  3oard  fssai 

3nd  contract  Data  Requirements  List  ( CDRL ) •  *  (5S3), 

01?*  Knowledge  of  Program  Assessment  Reviews  (PAR)  formand 
;?y^ent  ’(9vl5ws  ^AR)»  and  Program  Management  Requirement ’Transfer. 


% 

..  012.  Knowledge  of  QA  requ1rem°nts  needpd  fnr»  mi  *  o 

Management  Plan  (PMP),  and  (2)  Oevelooment  of  thpV^i  nt'  A  Pro^m 

( F  Sc  § ),  I  Sn°IJ; «  es  ’  Tf  shoil^cSv’  fU"  SCa,e  <f”0« 

cS"rP«"afQAQA^»?r6requireJtS>  ‘hr0^  td  of 

acceptance  by  the  Government.  « 

and  other °go ver mitent  Sor  ^ndus tr  v^f  a  !l  V° ^ V jm5n^ and  utilization  of  GIOE? 
involve  QA  activities  7  fa11ure  daU  exchan9e  Pr°9ra™  that  can 

cv<.fp_  nQ?-n‘  Understa"dl"9  Of  Che  QA  role  in  System  Requirements  Reviews 

FfA^  TnS  or?  ‘'evier?’  1mlnary  Oesign  Reviews,  Criticar Oesign  Reviews 
FCAs  and  PCAs  as  defined  in  rlIl-STD-1521 .  y  reviews, 

- .  .  Ability  to  olan  and  participate  in  PDR  CDR  Phvsiral  fnn 

nessrRev(e«Ud1tS’  FunctionaI  Configuration  Audits  and  Production  Read!-’ 

DAR  l-326°and  Pr0C5SS  for  »-»'—*•««"  of 

and  incentive  c^clcts^  PrePare  m°"';t0r  the  QA  Sections  °f 

ments  as  JequireJT  J^AFScV^r  ^  assessa,ent5  and  '^'^ndent  QA  assess- 

°l9'a.  Abl^^  t0  mar|aoe  implementation  of  those  portions  of  the  A F 

AFSC  J^uctrDivision°'1,P°"entS  C°'1tr°'  F’r°9ram  <ol’CC(,K'which  are  allocated  to 

•  • 

characteri*‘icsAfl^Ant0  critical  components  and  their  mandatory 

improvement  process  ““tefr^Srs^  °f  the  Prddu‘* 

021.  Ability  to  implement  and  negotiate  mutual  contract  understanding. 
Plans  thafare  program  Jn°iqueab'iSh’’  a"d  ”ai"tain  °A  pr°9rara  po,icies  a"d 


(OIs). 


023.  Ability  to  prepare  and  issue  QA  program  Operating  instructions 


project  officesAt>11ity  t0  provide  QA  coordin^ion  with  other  related  program/ 


o» 


KNOWLEDGES,  SKILLS  ir  abilities 
(KSAs) 

3. TECHNICAL  -  GENERAL  DISCIPLINES 


AN  MEASUREMENT  techniques 


nhues,  including^echan  leal, ^electronic3  opt  leal3"??  c°f  geaasureTent  tech" 

scales,  vernier  calipers;  optical  comparers, . voltage ^s',  2^5*. ’ 

A0  STANDARDS  of  workmanship  " x 

‘low  they  a?e‘deveTooidr^  ty  f0r  standards  of  workmanship, 

vsap  to^det ernfne  the^ccep tab U ityUof products ^  ^  lhd  A 

of  workmanship  standards'!  Ver'fy  the  adequaCy  of  contractor's  application 


/ 

S  *  0 


AP  iNTRODUCriON  TO  NDT 

test  of  n,au;,aU0WaIldtpp??esf!ndarenUI  C°"CeptS  °f  accept3"«  Action/ 

live  lnspec:ion/test!tand71asonsPforCuse!"S  ^  ,in"'utlons  °f  "ondestruc- 

radiography,  '.agSui^i'ieu*  "t^onkw^H^  pe^Int!""'^"9 
DRAWINGS  AND  SPECIFICATIONS 

p-'OdoctioS01'  Understandi"9  of  ose  of  drawings  and  specifications  in 

symbols ,  “oJjec??oiI;ydtaenrsS?on!no'n?n!in9  draw,"9s'  '"o'oding  lines, 
istrative  information.  -r3ncingf  parts  listed  and  admin- 

s-.andards!°^cl!!3!nge?o9rem!tf  ”11^  t""  FederaI  bonification  and 
changing  specifications.  ’  lna  ’on  procedures.  and  methods  of 

004.  Familiarity  with  000  Index  of  Specifications  and  Standards. 


METRIC  CONVERSION 


i s  s ues 


001.  Working 
surrounding  the 


knowledge  of  metric 
transition  to  metric 


system  and  familiarity  with 
measurements . 


s,s:m  J°L  PrimJry  unUs  °f  "easure  ,n  TOtr,= 

sjsiS-ZSnr'F”"””' as=“--sa“  ■“•■ 

-s  SASIC  STATISTICS 

001.  Understand  terms,  symbols  and  formulas  used  in  statistics. 

71923  ures  vi-f Sf oS a9** * 

cnar-ting.003,  Understand  fundamentals  of  process  control  statistics  and 

i*wf.  ’;s“?,ssg  a‘SsM 

;T  gSiSERVATIQN  ANO  PAEKTM: 

r*-!-tsd  t0  pneservation^packing’and  marking!"*  Spec1ficaUons  and  standards 

dacving  such *aSUcleaningd preservatives a^contlaintS  °f  preservation  and 
consols,  marking  and  labeling  conta^ers,  environmental 

packing  requlYe^  and 

AU  jELIAoILITY  AND  MAINTAINABILITY  , 

""  s  /  e 

re!  iabi  1  ityPtudies  ^and  thp^fh’n^  ,n  ^  10ns  and  concePts  used  in 
reliability  6S’  *"d  the  methods  used  in  Predicting  and  testing  for 

interpreting  sampl ing’andlest* t results  ‘  Uy  fr°m  Sampling  schemes  and 

of  -s  1  iabi  1  i ty  on  items  ^assembHes  ^  category  (minor,  .major)  the  effects 
cesi:n,  material  and  workmanship.  systems’  etc->  of  nonconformances  in 

-2inta{nab?h:tv!<stjdidr  °<nowledarof  and  concepts  used  in 

used  in  maintainability  studies  and  humaJ'factors ’dlPign?  ^  requirements 

* in^’ flJ. tSa^l"t!<!a^/o^,t-!^taa«h~.,nta,nab4 ’ 1ty 


10 


KNOWLEDGES,  SKILLS  &  ABILITIES 
(USAs) 

4  technical-specialized  disciplines 


COMPUTER  SOFTWARE 

AU  SOFTWARE  DESIGN 


.  00}\  Understanding  "top-down"  design,  structured  proora.cmino 

charts'*6  rn0du  es’  .des*9a  of  software  modules,-  Vee  structures  and  flew’ 

••n  nll\a  Skills  f;.reco9ni*ing  correct  design  decomposition  steps  and 
ln?  e]"r?nt  questions  concerning  individual  tree  structure  elenent- 

1; sUsJLn?  f,0“ chart  opera^,OTs 

a 

t i0n  TO^ies''md1ni°r^n'fare  thf .contractual  requirements  to  descrip. 

Mot:e-s  mrr'- and  S5»2SS'iis*^s 

skill  area)  b  y  "  Con”’uter  ero9r«,™<"g  (coding)  is  not  required  in  this 

“SO,  152  “aid  JESi'SiS'rSr.T-5-*?^.  0M-STO-9S0A,  HIL-S70S-433, 
design  d  a  0-179  (,lavy)  oogoirements  as  they  pertain  to  software 

oonly  used°°ii  AirnFo^e,appncat!onhsPr°9rOT  deSi9"  (?0Ls'  "m- 


ooog-ess 

*.  „  *  Al? i 1 1  to  maintain  technical  surveillance  ov«r  the 

5sl5Ji«  Snf?i'icaeinhas^C0-?iltKr  software  t00,s’  '“""'lies  and  metho- 

2^rSSE::rs«“- 


SOFTWARE  CODING  (PROGRAMMING) 


s  jf**are  -e-1*-nIJ^I!»i«n?iJ9  °f  ^  methodologies  used  for  translating 
!?■  2  ~  v'ea'J"  modules  into  a  programing  language*  CournonW  ik**  * 

'J^uL?-  7ort”n'  Jovi?1-  '’«=*'  md  Ada  (for  future” programs)  These 

i leases'  «r«^S?yS“!!n"i,^:1  S“t,,'8Mr’t'd  by  k"o-l.dge  of  m«hine 


I-?-s  S-c^P  constraints  ?n°?he  C SS^HLS^s?™!!^;^ 
in^the  use°of  KlTe^t,”  &£  ~  ~  >'*««*- 


compliance  with  the  contract.  (This  1«ve1  of  sfc11l~wiii  a[e  !n 

WTJ:JTC!r  Pr°m*  '  "or^J’th'e'iotTu^hJi'S,,,. 

■:.'<av„  as^y  and  MIl-STO-1579 

.ardS  and°progr  a^i  ng^convent  ions  "and^r  act  ices^utflized^byThlTcontr  actor" 


Aw  SOFTWARE  CONFIGURATION  CONTROL 


5.flratto?,2i«Si,i?o^s2;ta?  SpSSS?psft5r2°3a*;r*  ■rtt,,fc9"- 

conformance  or  bad  practices.  P  Sort,/are  and  in  recognizing  non¬ 

standard,  the  reason  foi^eachVule^or ^nrr/ow  eaC^  c°nf iguratlon  control 
tive  action.  rule  or  procsdurs’  and  specify  proper  correc - 

software  regoirements  as  they  pertain  to 

1 iprary  contro  Is  for  del  i verab  K* 'S  comPuter  program 
***  tile  computer  pro^ 


AX  IQ^^ARE  VERIFICATION  &  VALIDATION 


001.  Working  knowledge  of  methods  us°d  for  software  fa,f* 
^uooiog,  simulation,  em„,ation  and  V  1  V  (v^fiTaMoJ  Jn7 w“di?fen). 

compl iance^wit h^the 'contr actual ^requ irements ^fo^each5 test .rSSU^3  f0r 

dwierw!™.0?^’  to  '<>«‘<!  invalid  tests  and  improper  results  and 

functional  re,uirem^ts!rreCtneSS  0f  the  final  computer  Pfosram  to  satisfy 

to  software^ver  if?  cat  ior^and  ^iJiS”779  tS  “  - 

.  00^‘  Ability  to  understand  proper  methods  for  bcth  verifying  and 

bcth  deliverable  and  nondel iverable  computer  programs  and  asso- 
-ia«.ed  documentation. 


i ; 


COMPUTER  HARDWARE 


-  V  -DIGITAL  COMPUTER  EQUIPMENT 

001.  Working  knowledge  of  types  of  military  computers,  embedded 
vs  automatic  data  processing  systems,  computer  logic,  computer  languages, 
elements  of  3oolean  algebra,  arithmetic  operations,  storage  and  peripherals 

t  Skills  in  interconnecting  computer  elements  (hardware),  data 

oases,  jesting  systems  using  pre-programned  checkout  tapes  and  card  decks 
and  in  interpreting  results  of  system  checkout  prior  to  operation  use  of 
the  computer  hardware.  . 

,  Ability  to  understand  different  types  of  computer  hardware 

and  typical  use  and/or  limitations  in  actual  operation. 


AZ  MICROPROCESSORS  AND  FIRMWARE* 


001.  Working  knowledge  of  semiconductor  theory,  tyoes  of  "chiDs" 

’TJvr tPUt  H/0),  memory  capability,  RAM,  ROM,  PROM, 
-RDM  similarities  and  differences,  floppy  discs,  tapes,  drives,  displays. 


002.  Skills 
for  prooer  operation, 


in  Interconnecting  microprocessor  elements,  testing 
and  analyzing  results  prior  to  operational  use. 


003.  Ab 
firmware  "chips" 
operation. 


ility  to  recognize  different  types  of  microprocessors 
and  understand  their  uses  and  limitations  in  actual 


and 


BA  TECHNICAL  DATA  AND  PUBLICATIONS 

C01.  Knowledge  of  requirements  and  responsibilities  of  Quality 

°f  ««  I" 

-  ^ility  to  apply  Quality  Assurance  to  data:  00  Form  1423 

NeoaMvP  ?!?  Spe?in^ati°ns  and  Standards,  Artwork,  Photolithographic’ 
Caros  ’  n3lneering  Orawings,  Microfilms  and  Tabulating  and  Aperture 


E’-ECTRONICS/ELECTRICAL 
3B  BASIC  ELECTRICITY 


-,i  Knowledge  of  all  necessary  mathematical  concepts,  fundamon- 

dtre”nrfi?^S  electricity  including  sources  of  electricity,  magnetism’ 

4/  Mrr Ci*‘  ent  ■  tne^ry-  theories  and  uses  of  alternating  current  generation 
AC  circuitry,  impedance  and  resonance.  ^  ’ 


217,  MIL 


002.  Knowledge  of  associated  Specs 
-STD-750,  MIL-ST0-3S3,  MIl-S-19500, 


and  Standards  (i.e.  MIL-HC3K 
MIL-M- 38510,  MIL-STD-1313,  etc.) 
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3  C  BASIC  ELECTRONICS 


001. 


Of  theory  and  use  of  vacuum-tube  principles  and  circuit  ^Ihirh*  °  J*29* 

^?r,W 


o  D  MICROELECTRONICS 

,,T  .  ,  9?1*  ^owledge  of  hybrid  technology  and  terminoloov  thiek/‘hin 

ores^s^ooerat i'Qn-?rU  ^'  P?**'"!  °f  the  «S9  S 

hybrid  Circuits  and  ?e llibf T f SCa'“  ^  TO'I,U^,  sca,e  ,!,taSration, 


5E  DIGITAL  OEVICES 

iUndin, 

3  F  INTEGRATED  CIRCUIT  OEVICES 

linear  int egrat ed°c Ircuits f  **  °f  oparat,°"  of  b°tb  ^gital  and 

transistor^diode^an^resistance'logic)  storage^ lements  ftSW#d'B» 

™4JUS?l.*ta'"»  -(con^!°?ieTCnt3 . 

inspecti°" techni3ues  L 


3  G  RADAR  AND  MICROWAVE 


001.  Knowledge  of  wave  and  transmission  line 
crowave  components;  the  standard  radar  block  diagram; 


theory,  antennas  and 
individual  block 


U 


elements;  pulse  relationships;  types  of  radar  displays  and  various  types  of 
radar  systems;  and  radar  and  microwave  tests  and  measurements  and 
instruments  involved,  including  typical  specification  provisions 


3W  SUBSTRATE  TEST  MEASUREMENT 


001.  Knowledge  and  understanding  of  the  procedures  in  MIL 
and  the^  test  methods  in  MIL-STD-883  necessary  for  the  acceptance  of 
substrates  and  complete  microcircuit  devices. 


in  ruL-ri- oojIU 


002.  Familiarity  with  procedures  for 
and  quality  conformance  inspection. 

003.  Ability  to  interpret  inspection 
acceptance,  rejection  criteria. 


certif ication,  qualification 

v 

and  test  procedures  and 


AIRCRAFT  ANO  SPACE 

3  1  WEIGHT/BALANCE 


001. 

test  equipment 
MIl-W-25140. 


Knowledge  of  weight/balance  safety  regulations 
and  procedures  such  as  TO  1-1B-40,  TO  1-18-50* 


set-up  of 
AFR  81-6  and 


esults 


002. 


Ability  to  interpret  and  evaluate  weight/balance  test 


3J  FUEL/FUELING  SYSTEMS 

001.  Knowledge  of  fuels/fuel ing  system  theory  and  procedures- 
operation  controls;  fuel  line  installations  including  str1kinq°  kinkina  and 


results  . 


002. 


Ability  to  interpret  and  evaluate  fuels/fuel ing  systems  test 


- K  ENGINES  OPERATIONS  &  MAINTENANCE 

engine  op“lt ,J;"ow'ed9e  of  theory  and  Principals  of  gas  turbine  and  jet 


.f  now  on°?2,  Know1edge  Of  the  major  work  efforts  associated  with  the  OA 
e  en9  ne  procurements  including  turbojet  turbofan  turhnch;*«>  ►  , 

eng,V:  reUted  «««“?«  ’aJd 

■“noine  testino'anH V  a?d  e,ectr5ca1  requirements  associated  with  jet 

^  fnd  acceptance.  .he  specific  function  associated  with  jet 
iogine  accessories  and  components,  and  their  interrelationship  and  enqine 
SrQcfn,  on/performance.  Engine  buildup,  green  run  testing,  teardSwn  9 
procedures ,  inspection  methods  and  technique,  final  buildup,  final  accep- 
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rd  ?repf' >tlon  for  sh,«"‘- 

mentation  and  engine  p^far^nci  i.Ur.^Ipp^r^5'  teSt  docu- 

®  *"  -P-O'"^  PLA.NTS-FUELS  AND  LUBRICANTS 

light  distil iates^equ ?pmenttusedn?nf test ina'd is nnn?f^hyS^Ca^  properties  of 

specification  petroleum  producers . 

heavy  distiliates°Wproceduresausedtinyn|?^ofm?nt  Uued-,n  test1 ng  light  and 

laboratory  tests  and  interpreting  and  evalSauSg 

accomplishment  of  DoVorm^SO  ^ater i a V*" i|re'yS *  d<?ntractor  QA  programs, 
various  other  QA  responsibilities.  nspection  and  Receiving  Report  and 

4 

3M  HYPRAULICS/raEUMATICS 

c'oaed  re'“ed  t0  «* 

hydraulic  pump  and  motor  functions*  fHoht  109* proce<iur2S,* 

tionships  for  hydrauiic  actuators 

use  on  pneumatic  test(ng;J,pTOumaticWstritS0Je!-ati,draU,i5S  Snd  water  column 
control  for  canopy  seal  and  environmental  control  Jyste.ireUmaUC  SyStS!n 


3N  fM  4  POST-FLIGHT  AnflFPTANrr 

Of  a ircraf^includ'fng^fdm^l'i aNjatfon'^i^h^ the^t a"d  del1ver* 

services  which  prescribe  the  sDer-'fia-  technical  documents  of  the 

flight  and  post  flight  inspections-  arnfr!?  5  r°r  the  performance  of  pre- 
flight  tine  operation  which  nvSke^afetl  P™  T  fpHcadle  to  a 

of  aircraft,  use  of  supp„rt  e^Cent  JJ&.mITS1  f  «r,icinS 

ong.ne  operation  and  briefing’of^^t  c^s'p^ior 

Fl'9ht  and  engine  instruments  !"insta-l|faHonf?nrt°"nn9tarsas  ,s  des  i rah  1  e : 
and  subsystems  (except  electronic)*  and  a i -cr  f  chec,:0,Jtl  aircraft  systems 
subassemblies.  C  ’  and  aircraft  components  and 

tance  test  resuUs!Uy  t0  1nterpret  and  evaluate  pre  and  post-flight  accep- 


£0  EGRESS  SYSTEMS 

aircraft.001’  ,rnowled^e  of  Qualification  and  safety  criteria  need  for  the 


15 


orders . 


002.  Familiarity  with  applicable  military  standards  and  technical 


3P  MAINTENANCE  AND  OVERHAUL 

001.  Knowledge  of  various  types/models  of  aircrafts  and  the 
methods/procedures  by  which  shop  complete  and  final  inspections  of  contract 
items  are  conducted;  the  interrelationship  of  the  contractor  versus  PQA 
wor<  effort;  the  many  obligations  of  the  Government  such  as,  government- 
furnished  property,  aerospace  ground  equipment,  ground  support  equipment, 
special  tooling,  input  of  reparable  assets,  technical  documentation,  tech¬ 
nical  assistance,  work  statements,  procurement  specifications,  flight 
cs ts t  ferry  crews f  product  audits  and  work  request  proposals. 

Umi,  ,  „002-  knowledge  of  a  typical  M  &  0  operation  including,  but  not 

rec®1Pt/^hakedown. inspection,  shop  complete  inspection,  parts 
ep  acement,  rob  back  or  cannibalization  preflight  and  postflMqht 
inspections,  and  acceptance  and  delivery  of  the  aircraft,  contactor  QA 

p.ogram,  aircraft  workbooks,  over  and  above  activity,  and  flight  safety 
1  w  errs  • 


3Q  OXYGEN  ScR VICING/OPERATIONS 


w°!'  Knowledge  of  .theory  and  procedures  of  oxygen  servicing/ 
operations  including,  but  not  limited  to,  MIL -STD- 1 551 ,  QC  of  Gases  and 

12M01-  Genera'  S3fet' 


002.  Ability  to  interpret  and  evaluate 
operations  test  results. 


oxygen  servicing/ 


9  R  PROPELLANTS  AND  OXIDIZERS  (SPACE)  -  /. 

method*  o?°LiKn°Wledf  !n,the  use  of  laboratory  equipment  and  the  various 
methods  of  analyzing  alcohols,  esters,  aldehydes,  acids,  alkalies,  salts 

propellants  and  oxidizers  such  as  RP-1,  RJ-1,  hydrazine,  UDMH,  nigrogen 

tetroxide,  inhibited  red  fuming  nitric  acid,  liquid  oxygen  and  nitrogen. 

Application  of  approved  test  methods,  interpretation  of  test  results  and 
contractual  testing  requirements . 


3  S  ACCEPTANCE  TESTING  (SPACE) 


..  001  *  Knowledge  of  acceptance  testing  of  spacecraft  and  missiles 

and  the  procedures  that  must  be  followed  preliminary  to  and  during  the  per¬ 
formance  of  this  function;  the  specific  technical  requirements  as  pre¬ 
scribed  in  specifications  and  contract  requirements,  and  the  procedures 
appncaole  to  pretest,  during  test  and  post-test  requirements. 


002.  Certification  in  all  of  the  following  areas  is  desireable- 
spacecraft  and  missiles  systems  and  subsystems  (except  electronic);  and 
spacecraft  and  missile  airframe  components  and  subassemblies. 
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MANUFACTURING  PROCESSES 


.8  T  AOHSSIVE  BONOED  STRUCTURE 

001.  Knowledge  of  characteristics,  procedures,  techniques,  appli¬ 
cations  and  limitation  of  adhesives,  solvents  and  potting  compounds  for 
mechanically  joining  parts  of  structures. 

002.  Familiarity  with  advantages  and  disadvantages  of  using  adhe¬ 
sives  as  a  joint  material;  types,  characteristics  and  uses  of  different 
adhesives;  joint  designs,  bonding  techniques  and  preparations  necessary  for 
a  properly  bonded  joint;  controls  of  adhesive  materials  prior  to  and  during 
use;  inspection  and  testing  technqiues  for  suitability  of  joint  or 
assembled  bonded  structure  and  techniques  for  repair  of  defective  bonded 


003.  Knowledge  of  materials  and  processes  used  in  construction 
and  inspection  of  composite  structures  such  as  graphite,  bor$n,  and  glass 
fibers,  honeycomb  structures,  layup  and  curing  techniques,  vapor  deposi¬ 
tion,  and  test  and  repair  techniques. 


3  U  PLATING 

001.  Knowledge  of  the  theory  and  procedures  of  plating. 

002.  Knowledge  of  the  chemistry  of  plating,  ion  vapor  deposition 
fiu  e^ct’-oplating,  nickle  electroplating,  chromium  electroplating  and 
other  types  of  plating.  y 

i.-  verifying  the  adequacy  of  contractor  plating  opera¬ 

tions  and  the  quality  of  the  item  being  inspected. 


8  V  CLEANING  „  /. 

..  °?1*  *no*led9e  of  the  purposes,  equipment,  materials  used,  pro- 

of  rcatfflal  (^equipment ^  C'ea,”n9  °f  neW  ^  retUri,ed  fn”  Serv,ce  its»s 

electrica^methodj1  of  Cleaning!!  Ch8"f<:a1,  S°I'"!nt’  •  s0"lc 


8  W  WELDING 

„ai1.  00i‘  knowledge  of  concepts  associated  with  weld  inspection:  sur- 

ve il lance  of  on-going  QA  procedures  in  welding  facility;  and  the  visual 
non-destructive  inspection  and  mechanical  testing  of  we  ded  structures  * 
(fusion,  spot/seam,  and  other  welding  processes);  structures 

0°2-  Ability  to  determine  acceptability  with  regard  to  the 
specified  quality  level. 


BX  HEAT  TREAT 


cesses 


,  001-  Knowledge  of  equipment  and  operations  of  the  heat  treat  pro 
*  or  i^rrous  and  aluminum  alloys.  y 


002.  Familiarity  with  the  coomon  heat  treatment  processes,  effects 
of  improper  procedures,  controls  and  test  methods  needed  to  determine  com- 
plianca  with  raquiremants ,  methods  of  heating  materials,  principles  of  harden¬ 
ing  and  softening  metals  by  heat  treatment,  and  calibration  of  equipment. 


B  Y  SOLDERING 

001.  Knowledge  of  present  techniques  and  requirements  for  soldering. 

002.  Fami 1 iarity  with  current  inspection-  procedures ,  proper  tech¬ 
niques,  tools  and  materials  according  to  the  latest  standards  for  soft 
soldering  electrical  connectors.  > v 

003.  Ability  to  accept  or  reject  soft  soldered  electrical  connec¬ 
tions  according  to  pertinent  specifications. 

BZ  ELECTROPLATING  AND  AN0DIZIN6 

001.  Knowledge  of  the  equipment  and  procedures  related  to 
electroplating  and  anodizing. 

002.  Familiarity  with  the  purposes,  preparation  of  base  metal, 
processes  and  controls. 

003.  Ability  to  test  the  plated  and  anodized  surfaces  to 
specification. 


CA  FINISHES.  PAINTS  AND  SURFACE  TREATMENT 

001.  Knowledge  of  the  various  surface  finishes  applied  to  metals 
and  metallic  products  for  protection. 

s  /  * 

002.  Familarity  with  the  materials  used  as  finishes,  the  pro¬ 
cesses  of  applying  the  finishing  materials  including  painting,  black  oxide 
coating,  phosphate  coatings,  chromium  conversion  coatings,  and  passivation 
of  stainless  steels. 

003.  Ability  to  perform  tests  required  to  accept  or  reject  per 
specifications . 


CB  CORROSION  CONTROL 

001.  Knowledge  of  the  equipment  and  procedures  used  in  corrosion 
abatement . 

002.  Familiarity  with  the  types  of  corrosion  such  as  water 
intrusion,  galvanic,  vapor  corrosion,  etc. 


CC  BRAZING 

001.  Knowledge  of  the  various  processes  used  in  brazing. 
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speclfi cations. <b,"ty  *°  aCC'pt  °r  reJsct  ljra"d  J°'"“  P*'  Applicable 


CD  OPTICAL  TOOLING  AND  ALIGNMENT 

001.  Knowledge  of  application  of  optical  tooling  and  alignment, 
problem  encount ered'when^usi ng  optical  J"d  SpWi"3 

003.  Ability  to  operate  and  checkout  optical  tooling  equipment. 
CE  DIMENSIONS  AND  TOLERANfTS  • 

with  uniform' interpretation  Sethis  ?«hnSlog™  dr*"'n3s  tec,,n°,°S*  Along 
002.  Ability  to  read  and  interpret  drawings. 

C  F  LASERS 

ooi.  Knowledge  of  laser  characteristics  and  various  applications. 

log  appmat2ionsn?uchdLe  Includ- 

treating  irregular  geometries  »d I  rws  h«' 

CIS  ion  spot  welding,  lonq  stand  nff  Woiw<’  tw?.sided  bimetal  welding,  pre- 
high-yeild  hermeUc«U  „  ‘  « k *?,  h^oeT*  ^°nt0ured 
distortion-free  cutting,  ha^d  plated  met !l  °ra"<d''-«:tional 

partial  cutting,  engravinq  refrae*nrlnn«  9*  d?ep  sect,on  Cutting, 

high-volume  cutting.  *  ^  non-metal  cutting  and  non-metal 


:G  FERROUS  AND  NON-FERROUS  METALS 

shaping,  and  treating^?  metals  fncludinqVxtr  p[?cesses  !n  the  making, 
forming  and  heat  treating  of  metals  and  nr«i-raCt  0n’  r®flnin9i  alloying, 
used  in  metal  production^  manuflct^rinS  theS0^iSir°  S-  f6st  methods 
the  cycle  from  basic  metal  to  final  product.  9  n  °r  metal  defects 

cal  tests  cannon ly  required  b^contraci5  including  of. conducting  mechani- 
shear,  cold  bend,  harness  and  5mpac?  tesUng  109  te"Sl1e’  COmpressi'°". 


NO ; /NO I  AND  ENVIRONMENT  TESTING 
CH  RAQIQGRAPHY 


001.  Knowledge  of 
radioactive  isotopes. 


the  theory  of  radiography. 


including  x-rays  and 


002.  Knowledge  of  the  applications  and  limitations  of  test 
methods,  application  of  specifications  and  standards  pertaining  to  radio- 
graph ic  inspection.  r  * 

003.  Ability  to  make  radiographic  set-ups,  positioning  and 
operating  radiographic  equipment,  and  processing  and  interpreting  films. 


004. 

operations  and 
inspected  through  radiographic  techniques. 


Skill  in  verifying  adequacy  of  contractor  radiographic 
the  quality  of  the  items,  including  microcircuits  being 


C  I  MAGNETIC  PARTICLE 

001.  Knowledge  of  magnetic  particle 
procedures  and  techniques. 


inspection  and  magnetizing 


002.  Ability  to 
tion  requirements. 


interpret  test  results  and  pertinent  specifica- 


C  J  LIQUID  PENETRANT 

001.  Knowledge  of  liquid  penetrant  inspection  procedures, 
applications!^  Ca  pr1nc1p1es*  Pre-deaning,  dwell  time  and  fluorescent 

tion  requirements !"ty  t0  ,nterpret  test  resuUs  and  Pert,"«"t  specific*- 

c  K  ultrasonics 

.nH  fo,hn5n01.  Knowledge  of  ultrasonic  inspection  principles,  procedures 
and  techniques  or  applying  ultrasonic  energy  to  the  Inspection  of  materials 

betweeJematerialsth!ndrfornthirk  discontinui’ties  material,  lack  of  bond 
ei  een  materials  and  for  thickness  measurement. 

transmiss?on*n^nlfr!d9ei0f  the°ry  °f  u1trasonic  energy,  the  generation  and 
”  sl  o.  ultrasonic  energy,  couplant  and  transducer  theory  and 

rnnfU?  ’  a *w  det?ctl0n  and  thickness  measurement  applications,  surface 
contact  and  immersion  techniques. 

tier  requirements ! ' t0  ,nterpret  t.est  resuUs  Pertinent  specific*. 


CL  LASER  BONO  INSPECTION 

001.  Knowledge  of  laser  bonding  procedures  and  theory. 

*  ?02*  Knowledge  of  solid  state  and  gas  filled  lasers,  lase*-  optics 

'ondpiTa  insta,’a^ion-*  .bond  inspection  principles  includino  locating  un-  ' 
:  d  areas  and  determining  unbonded  bounderies,  detecting'void  in 

if  «r  onne|9h^LeH,bea"'nSPrek'f;Sht[t  techniques,  end  thorough  knowledge 
1  Is”  iiui^eit  35  r  3  Ur"S  a"d  l’1gh  vo,u5e  shocks  in  using 
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CM  ADHESIVE  BONO  INSPECTION 


S'" 


mechanically  Joining  parts  of  structures. 


•  bonded  Sisn-cSSi?;  irajs^^KK'iiic  f 


si  ves 
adhes 

a  properly  bonded  joi^'c^il  .T^SlJelStKK'SilS  to  SV  <* 
use;  inspection  and  testing  techniques  for  sultabill?!  nf  «<  l  d  dur1ng 

areas^  6d  ■>—  «"««"?  -  t^,JS  bonded 


C  N  EDOY  CURRENT 

techniques0''  >inow,5d9e  of  edd>  Inspection  procedures  and 

discontinuities  in  metals^includlnq^such  items3.^  surf?5e  and  subsurface 
machined  ££“  °f  hd’“  ^ ‘Se“S«J’^S,|&.,“ta,#,“* 

interpret°?«t  Results /°ndP!ccepteo?yrejectnpartsU!Pment'  r“d  and 


Ij-ECTROMAGNETIC  INTERFERENCE  CPMPATTRTt  rtv 
patibility  requirements6and  testing!*^  pr0Cedures  of  electromagnetic  com- 

S 

monitor  test' ’and' Tell dement 'programs  relltivrt^radfati60^ ' analysis  to 
interference,  pollution  and  hazards*  mathematic?  ^dlJJ1on  fluency 
power  ratios  (decibels)*  thecrv  and’..**  <  ,cons derations  including 

such  as  RF  sniffers  soictrSTJni?^  yPka1  Be»*ur««nt  equipment 

meters;  various  technlq^s  ^lau'f to^o^s^'anfnl^  Tt  *"'*» 

cents  and  the  compilation  of  data  and  the  recording  of  dm?  * 

cp  environmental  test 
001 


and  high 
dust,  sa 
tion  and 


Knowledge  of  theory  and  procedures  of  environmental  testing, 
temper  at  ure  ^hu-nfriTt  ^  var1ous  environmental  tests  including  low 


003. 


Ability  to  accept  or  reject  test  results  per  specifications, 
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MEASURING  ANO  TEST  EQUIPMENT 

.  CQ  ’ELECTRONIC  MEASURING  EQUIPMENT 

001.  Knowledge  of  the  theory  of  electronic  measuring  equipment. 

002.  Ability  to  use  all  types  of  current,  voltage,  and  resistance 
indicating  devices  such  as  meters;  power  supplies  and  regulators;  resist¬ 
ance  and  impedence  measurement  equipment;  waveform  display  and  analysis 
equipment;  signal  generators;  and  frequency  measurement  Instruments  and 
devices  Including  meggers,  Hl-Pot  testers,  capacitance  testers,  transistor 
testers,  and  tube  testers. 


CR  MECHANICAL  MEASURING  EQUIPMENT 

001.  Knowledge  of  mechanical  measuring  equipment  techniques  and 
the  attendant  mathematical  requirements.  « 

002.  Ability  to  use  precision  measuring  equipment  Including  the 
techniques  and  setups  required  to  obtain  the  measurement  specified  by  the 
drawing. 


003.  Ability  to  use  equipment  such  as  surface  plates,  thread 
measuring  equipment,  optical  comparators,  scales,  micrometers,  vernier 
calipers  and  height  gauges. 


C  S  AUTOMATIC  TEST  AND  MEASURING  EQUIPMENT 

001.  Knowledge  of  automatic  test  and  measuring  equipment  and  the 
programming  of  such  equipment. 

002.  Ability  to  use  various  automatic  test  and  measurlnq 

equipment.  ,  /, 


CT  SCANNING  ELECTRON  MICROSCOPE 

001.  Knowledge  of  scanning  electron  microscope  procedures  and 
techniques . 

002.  Ability  to  use  scanning  electron  microscopes  as  an  Inspec¬ 
tion  too*  and  the  application  of  acceptance  and  rejection  criteria  and 
decision-making  techniques. 


LU  INTERRELATED  TECHNICAL  DISCIPLINES 

001.  Knowledge  of  basic  reliability  and  maintainability  program 
requirements  and  demonstration/test  (includes  familiarity  with  MIL-STDS-785 
470,  471,  and  731). 

002.  QA  requirements  as  applied  to:  (1)  nuclear  hardened 
systems;  and  (b)  QA  requirements  and  tests  that  may  be  applied  to  systems 
or  equipment  that  must  be  survivable  or  systems  required  to  operate  in  an 
electronic  Warfare  (E-W)  environment. 
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KNOWLEDGES,  SKILLS  &  ABILITIES 
(KSAs) 

5  MANAGEMENT  OISCIP1 INFS 


CU  BASIC  SUPERVISORY 

nel  po  1  1c°es  *  anl^procedures?  °f  a"d  abi"ty  to.compl1'  ",th  ‘<*"1*"  Person- 
enployee  Mrfor^ci^o^hSe' ,f  Perf°™ance  apd 
evaluation°inter views*  t0  C°ndU'Ct  e""”0J™ent  ,nt“-'',e"s  and  performance 

and  superv  is  ory^responsibiMties^re  la  ted^  thereto?"9 'dS  «  lhe  £E°  prP*ra"" 

Core  and  S^tems^DIslipl^ 'lnM)°Md“bll urtolpply!"*  c0"cepts-  (Terence 
System  Disciplines)!^*  t0  c<”u,’,cate  Actively-.  (Reference  Core  and 

projects.0!  other' group-part icipativ^tasks!1*1176  ^  fPr« 

PerformancTif  a"d  - 


CV  INTENSIVE  SUPERVISORY/MANAGEMENT 


Of  autnor?!!"^!^"?,?;.3!3!!^  •jrS'iSSK*,"  d^atl°" 

carry  through  efH^X'^WR)?'^9  Se“'"9  C°"CePtS’  ^  abili:'  tP 

003.  Ability  to  understand  and  capacity  to  choose  Pth<rai  i„„d 
cal  and  practical  actions  in  relation  to  J^  foecis!  ,!!^)  9 

-oraers.  supervisors ^n^subord inates? '*’  “d  Creatively- 

and  superior!  Ab"Uy  t0  mot,vatl!  and  inf,UI!"«  employees  both  subordinate 

..  .°06,  Understanding  of  essential  elements  of  effectina  chanap  anH 

the  ability  to  make  changes  efficiently  and  smoothly.  9  Change’  and 
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C T  r» 


007.  Understanding  of  principles  of  time  management  and  the  abi¬ 
lity  to  practice  the  concepts. 


CW  ADVANCED  MANAGEMENT  CONCEPTS 

001.  Understanding  of  program  management  organizational  struc¬ 
tures  jnd  analyses  (functional,  project,  matrix,  combinations) . 

002.  Understanding  of  theory,  organization  and  design  of  manage¬ 
ment  systems,  and  the  concept  of  viewing  organizations  as  a  total  system. 

003.  Understanding  of  Management  Information  Theory,  including 
the  generation,  organization,  transformation,  dissemination,  codification, 
discrimination  and  economics  of  information. 

004.  Understanding  of  managerial  applications  of  automated  data 
systems  with  emphasis  on  management  information  systems.  J 

005.  Ability  to  make  effective  decisions  based  on  relevant  fac- 
ors,  including  statistical  theory  and  techniques,  matrix  charting,  flexi- 
ility  and  originality  in  thinking,  effective  listening  habits,  etc. 

006.  Understanding  of  the  relevance  and  significance  of  beha¬ 
vioral  and  social  sciences  regarding  human  organizational  behavior. 

007.  Understanding  of  Productivity  concepts  and  applications. 

008.  Understanding  of  the  essential  elements  of  leadership, 
including  specific  behavioral  techniques,  analysis  of  reactions  of  others 
group  dynamics,  etc.  ’ 


25 


APPENDIX  C 


TABLES  OF  TASKS,  RESPONSIBILITIES,  AND  TRAINING  AREAS 


TABLE  OF  RESPONSIBILITIES  BY  TASKS 
WITH  I  or  II  RANKING  IN  TABLE  FIVE 
(See  Table  Four  for  meaning  of  Roman  Numerals) 
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15.  AFPRO  or  DCAS  has  responsibility  to  II  II  I  I  —  —  III 

finalize  preaward  report  and  provide 
the  report  to  the  SPO. 


18.  Obtain  general  assistance  and  II  II  I  II 

information  on  contractor’s  general 

capabilities  in  software  areas  from 
AFCMD.  Augment  AFCMD  capability  with 
SPO  personnel  during  conduct  of  the 
preaward  survey. 

19.  Establish  channel  of  communication  I  I  I  I 

between  SPO  and  AFCMD  specific  to  new 
program/project. 


Ill 


21.  Conference  contract  administration,  I  II  II  II  —  —  n 

quality  assurance,  engineering,  and 
other  AFCMD  support  as  necessary  to 
ensure  flow  of  information  to  SPO. 


22.  Identify  contract  requirements  for  I  I  —  III 

performance  measurements. 

23.  Define  the  relationships  and  I  I  I  I 

responsibilities  of  the  SPO  and  AFPRO 

for  the  contract,  including  a 
memorandum  of  agreement,  if 
appropriate. 
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24.  General  review  of  draft  RFP  for  I  I  III  HI  — 

consideration  of  all  components  to 
ensure  enforceability  of  contract 
requirements,  proper  subcontract  or 
management,  complete  pricing 
requirements,  and  consideration  of 
delegation  of  some  monitoring  efforts 
to  AFPRO  by  SPO  not  already  included  in 
the  RFP. 


25.  Review  of  the  draft  SOW  to  determine  I  I  III  III  --- 

whether  the  management  requirements, 
software  development  of  criteria, 
procedures,  referenced  to  CDRLs  and 
specifications,  and  other  consideration 
provide  adequate  means  to  properly 
monitor  the  contract. 


26.  Review  of  the  draft  CDRLs  to  deter-  I  I  III  III  —  —  III 

mine  the  effectiveness  of  the  DIDs  and 
detailing  of  CDRLs  provide  adequate 
means  to  properly  monitor  the  contract. 
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27.  Review  of  the  draft  RFP  to  determine  I  I  III  HI 

the  reasonableness  of  the  overall  time 

duration;  whether  the  delivery  schedule 
for  various  plans  supports  their  valid 
useability;  identification  of  critical 
needs  for  long  lead  times;  appropriate¬ 
ness  of  milestones  and  sequences;  areas 
of  high  potential  risk  to  schedule 
maintenance. 

28.  Review  of  the  draft  RFP  to  determine  I  I  III  III 

the  presence  of  appropriate  and  current 

versions  of  standards  and 
specifications  and  to  check  tailoring 
against  requirements  for  valid 
administration  of  contract. 


Ill 


29.  Prepare  and  review  sample  contract  to  I  I  III  III 
ensure  inclusion  of  necessary  clauses 
and  forms,  deletion  of  unnecessary 
forms  and  clauses,  proper  cross 
references  to  MIL-specs  included  in 
SOW/CDRL,  and  other  technical  and 
clerical  details. 


Ill 
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48.  Review  contract  as  awarded  to  determine 
presence  of  software  functional  areas 
such  as  Work  Breakdown  Structure, 
Military  Standards  and  Specifications, 
Contract  Data  Requirements  Lists, 
Software  Development  Plan,  Software 
Configuration  Management  Plan,  Software 
Quality  Assurance  Plan. 


49.  Identify  and  document  potential  I  I  III  II  --- 

contract  administration  problems  due  to 
terms  and  conditions  of  the  contract. 


50.  Insure  that  formal  agreements  (such  as  I  I  I  I  — 
Memorandum  of  Agreement)  exist  between 
SPO  and  CAS  identifying  their  specific 
duties  and  responsibilities  concerning 
software. 


51.  Develop  an  internal  CAS  plan  for  I  I  III  III  —  —  HI 

assignment  of  manpower,  test  and  review 
participation,  inspection  points,  and 
communication  responsibilities. 
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52.  Identify  and  develop  plan  for  desired  II  II  —  II  —  —  I 

contract  administration  activities  for 
subcontracted  software. 


53.  Optional  SPO  and  CAS  Postaward  I  I  III  II 

Orientation  Conference  of  software 

concerns  and  responsibilities. 

54.  Ongoing  evaluation  and  review  of  I  I  m  m 

contractor’s  management  system  for 

software  development. 

55.  Monitor  critical  subcontracts  for  flow-  II  II  III  III 

down  of  software  contract  requirements. 


II 


III 


I 


56.  Monitor  cost,  schedule,  and  performance  I  III  III  III 
of  software  development  tasks. 


Ill 


57.  Resolve  technical  and  management  I  II  III  III  —  —  III 

problems  in  software  development 
process. 


58.  Review  and  evaluate  software 
Engineering  Change  Proposals. 


I  III  III  III 


III 
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59.  Review  contract  modifications  for  any 
possible  effects  on  monitoring  software 
development. 


60.  Monitor  development  and  deliveries  of 
software  Contract  Data  Requirements 
List  documents. 


61.  Monitor  and  review  compliance  with  I  II  III  —  — 

automated  test  system  standards  and 
documentation. 


62.  Audit  contractor  compliance  with  II  —  —  — 

software  requirements  for  configuration 

management  and  quality  assurance. 

63.  Review  and  report  to  SPO  on  con-  I  m  —  —  — 

tractor’s  technical  planning  and 

management  of  software  development 
prior  to  each  formal  review. 

64.  Follow  up  all  action  items  designated  I  I  —  —  — 

by  SPO  as  a  result  of  each  formal 


review. 
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67.  Assess  the  contractor’s  system 

development  tools  and  management  and 
technical  abilities  to  carry  out  the 
necessary  development  functions. 


I  III 


68.  Review  the  Computer  Software 

Configuration  Item’s  functional, 
performance,  data  base,  qualifi¬ 
cation,  interface,  and  delivery 
requirements  through  the  formal 
Software  Specification  Review. 


I  II . 


69.  Assess  proceeding  with  contractor’s 
designs  for  each  Computer  Software 
Configuration  Item  through  the  formal 
Preliminary  Design  Review. 


I  I  — 


71.  Monitor  software  coding  phase. 


I  II 


72.  Monitor  software  integration  and  test.  i  n  _  _ 


PROPERTY  MANAGEMENT o 
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76.  Document  the  completeness  of  meeting  II  I  —  —  — 

requirements  and  specifications,  and 

testing  of  each  Computer  Software 
Configuration  Item  and  Software  Product 
through  formal  Functional  and  Physical 
Configuration  Audits. 

77.  Monitor  planning  and  execution  of  I  II  —  — 

Independent  Verification  and 

Val idation. 


78.  Monitor  development  of  automated  tools 
used  to  prepare  technical  manuals, 
operating  procedures,  engineering 
drawings,  etc.,  to  assure  compatibility 
with  end-user  systems. 


79.  Monitor  assurance  that  configuration  I  II  —  — 

management  and  historical  data  systems 
are  maintained. 
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80.  Determine  capability  of  maintenance  I  II  —  —  — 

procedures  to  human  factors  and 
equipment. 


81.  Evaluate  training  requirements  for  I  II  —  —  — 

operation  and  support  of  software 

system. 

82.  Evaluate  software  system  document-  I  HI  —  —  — 

ation  to  assure  support  of 

postacceptance  maintenance. 

83.  Evaluate  logistical  support  to  system  1  111  —  —  — 

users  for  postacceptance  maintenance. 


84.  Complete  software  acceptance 
procedures . 
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85.  Review  the  Configuration  Management 
Plan  for  postdevelopment  software 
support. 


86.  Review  the  Software  Quality  Assurance  I  I  —  —  — 

Plan  for  postdevelopment  software 
support. 


87.  Review  Software  Test  Plans  for 
postdevelopment  software  support. 


88.  Determine  that  proposed  engineering  I  II  —  —  - 

changes,  deficiencies,  and  latest 
defects  in  software  are  corrected  by 
the  contractor  and  the  Configuration 
Management  Plan  adjusted  accordingly. 


89.  Establish  a  Configuration  Management  I  II  —  —  — 

Plan  for  postdevelopment  software 
support. 
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22.  Identify  contract  requirements  for  X 

performance  measurements. 


X  X 


24.  General  review  of  draft  RFP  for  X 

consideration  of  all  components  to 
ensure  enforceability  of  contract 
requirements,  proper  subcontract  or 
management,  complete  pricing 
requirements,  and  consideration  of 
delegation  of  some  monitoring 
efforts  to  AFPRO  by  SPO  not  already 
included  in  the  RFP. 


X  X  X  X  X 


X  X 


25.  Review  of  the  draft  SOW  to  determine  X  XX 

whether  the  management  requirements, 
software  development  of  criteria, 
procedures,  referenced  to  CDRLs  and 

specifications,  and  other  consid- 

/ 

eration  provide  adequate  means  to 
properly  monitor  the  contract. 
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26.  Review  of  the  draft  CDRLs  to  deter¬ 
mine  the  effectiveness  of  the  DIDs 
and  detailing  of  CDRLs  provide 
adequate  means  to  properly  monitor 
the  contract. 


27.  Review  of  the  draft  RFP  to  determine  XX  X 

the  reasonableness  of  the  overall 

time  duration;  whether  the  delivery 
schedule  for  various  plans  supports 
their  valid  useability;  identi¬ 
fication  of  critical  needs  for  long 
lead  times;  appropriateness  of 
milestones  and  sequences;  areas  of 
high  potential  risk  to  schedule 
maintenance. 

28.  Review  of  the  draft  RFP  to  determine  X  X  X  X  X 

f 

the  presence  of  appropriate  and 
current  versions  of  standards  and 
specifications  and  to  check  tailor¬ 
ing  against  requirements  for  valid 
administration  of  contract. 
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29.  Prepare  and  review  sample  contract  X  X  X  X  X  X  X 
to  ensure  inclusion  of  necessary 
clauses  and  forms,  deletion  of 
unnecessary  forms  and  clauses, 
proper  cross  references  to  MIL- 
specs  included  in  SOW/CDRL,  and 
other  technical  and  clerical 
details. 

48.  Review  contract  as  awarded  to  deter-  X  XX 

mine  presence  of  software  functional 

areas  such  as  Work  Breakdown 
Structure,  Military  Standards  and 
Specifications,  Contract  Data 
Requirements  Lists,  Software  Devel¬ 
opment  Plan,  Software  Configuration 
Management  Plan,  Software  Quality 
Assurance  Plan. 

/ 

49.  Identify  and  document  potential  X  X  X  X  X  X  X 

contract  administration  problems  due 

to  terms  and  conditions  of  the 
contract. 
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51.  Develop  an  internal  CAS  plan  for  X  XXX  X 

assignment  of  manpower,  test  and 
review  participation,  inspection 
points,  and  communication  responsi¬ 
bilities. 


54.  Ongoing  evaluation  and  review  of  XXX 

contractor’s  management  system  for 

software  development. 

55.  Monitor  critical  subcontracts  for  X  XX  X 

flow-down  of  software  contract 

requirements . 

56.  Monitor  cost,  schedule,  and  perform-  XXX  X 

ance  of  software  development  tasks. 


57.  Resolve  technical  and  management  XX  X 

problems  in  software  development: 

process . 

58.  Review  and  evaluate  software  XX  XX  X 

Engineering  Change  Proposals. 
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59.  Review  contract  modifications  for  XX  XX 

any  possible  effects  on  monitoring 

software  development. 

60.  Monitor  development  and  deliveries  XX  XX 

of  software  Contract  Data  Require¬ 
ments  List  documents. 

61.  Monitor  and  review  compliance  with  X  XX 

automated  test  system  standards  and 

documentation. 


X 


X 


X 


62.  Audit  contractor  compliance  with  XX  X 

software  requirements  for  configur¬ 
ation  management  and  quality 
assurance. 


63.  Review  and  report  to  SPO  on  con¬ 
tractor’s  technical  planning  and 
management  of  software  development 
prior  to  each  formal  review. 


X  X  X  X  X 
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64.  Follow  up  all  action  items 

designated  by  SPO  as  a  result  of 
each  formal  review. 


67.  Assess  the  contractor’s  system 

development  tools  and  management  and 
technical  abilities  to  carry  out  the 
necessary  development  functions. 


68.  Review  the  Computer  Software  Confty-  X 
uration  Item’s  functional,  perform 
ance,  data  base,  qualification, 
interface,  and  delivery  requirements 
through  the  formal  Software  Specifi¬ 
cation  Review. 


69.  Assess  proceeding  with  contractor’s 

designs  for  each  Computer  Software 

/ 

Configuration  Item  through  the 
formal  Preliminary  Design  Review. 


71.  Monitor  software  coding  phase.  X 


X  X  X 
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72.  Monitor  software  integration  and 
test . 


X  X  X  X  X 


76.  Document  the  completeness  of  meeting 
requirements  and  specifications,  and 
testing  of  each  Computer  Software 
Configuration  Item  and  Software 
Product  through  formal  Functional 
and  Physical  Configuration  Audits. 


X  X  X  X  X 


X 


77.  Monitor  planning  and  execution  of  X  X  X  X  X 

Independent  Verification  and  Valid¬ 
ation. 


78.  Monitor  development  of  automated  XX  X 

tools  used  to  prepare  technical 

manuals,  operating  procedures, 
engineering  drawings,  etc.,  to 
assure  compatibility  with  end-user 
systems. 

79.  Monitor  assurance  that  configuration  x  X 

management  and  historical  data 

systems  are  maintained. 


X 


X 


Dennis  Blosser  &  Associates 
Final  Report 
Page  Appendix  C - 18 


(Continuation) 

TABLE  OF  TRAINING  TOPIC  AREAS 
FOR  ENGINEERING  TASKS 


fc# 

-J  LLl 
OS 
ZD  m 
arzD 
c r 


Ui 

-j 

>- 
■  o 


CD  . 

«c 


:  o 
:  to 


CL 

=  CO  to 

— J  UJ 

C  to  LU  ZD 

O  CO  h  O ' 

OC  to  c  HH 

zo>-  a  r 

3=  »—  -J  |—3= 

C_)  C_>  c  Ll_  CJ 

LU  <  Z  O  LU 

hL<  tO  I— 


<  O 

:  to 


cm  e 
<  c 
:■ - 

t: 

O  L 

to  C 


CC  ( 

5 1 
fc! 

O  1 

to  ( 


80.  Determine  capability  of  maintenance  XX  XX  X 

procedures  to  human  factors  and 

equipment. 

81.  Evaluate  training  requirements  for  X  X  X  X  X 

operation  and  support  of  software 

system. 


82.  Evaluate  software  system  document-  X  X 

ation  to  assure  support  of  post¬ 
acceptance  maintenance. 

83.  Evaluate  logistical  support  to  X  XX 

system  users  for  postacceptance 

maintenance. 


84.  Complete  software  acceptance  pro¬ 
cedures  . 

/ 


X 


X  X  X  X  X 


X  X 


85.  Review  the  Configuration  Management  XX  XX 

Plan  for  postdevelopment  software 
support. 
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86.  Review  the  Software  Quality  X 

Assurance  Plan  for  postdevelopment 
software  support. 

87.  Review  Software  Test  Plans  for  post¬ 
development  software  support. 

88.  Determine  that  proposed  engineering 
changes,  deficiencies,  and  latest 
defects  in  software  are  corrected  by 
the  contractor  and  the  Configuration 
Management  Plan  adjusted  according¬ 
ly. 


XXX  XX 


XXX 


XXX 


89.  Establish  a  Configuration  Management  X  X  X  X  X  X  X 

Plan  for  postdevelopment  software 
support . 
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22.  Identify  contract  requirements  for 
performance  measurements. 


X  X  X  X  X 


24.  General  review  of  draft  RFP  for  XXX 

consideration  of  all  components  to 
ensure  enforceability  of  contract 
requirements,  proper  subcontract  or 
management,  complete  pricing 
requirements,  and  consideration  of 
delegation  of  some  monitoring  efforts 
to  AFPRO  by  SPO  not  already  included 
in  the  RFP. 


25.  Review  of  the  draft  SOW  to  determine  X  X  X  X  X 

whether  the  management  requirements, 
software  development  of  criteria, 
procedures,  referenced  to  CDRLs  and 
specifications,  and  other  consid¬ 
eration  provide  adequate  means  to 
properly  monitor  the  contract. 
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26.  Review  of  the  draft  CDRLs  to  deter-  x  XXX 
mine  the  effectiveness  of  the  DIDs 
and  detailing  of  CDRLs  provide 
adequate  means  to  properly  monitor 
the  contract. 


27.  Review  of  the  draft  RFP  to  determine  x  XXX 
the  reasonableness  of  the  overall 
time  duration;  whether  the  delivery 
schedule  for  various  plans  supports 
their  valid  useability;  identi¬ 
fication  of  critical  needs  for  long 
lead  times;  appropriateness  of 
milestones  and  sequences;  areas  of 
high  potential  risk  to  schedule 
maintenance. 


28.  Review  of  the  draft  RFP  to  determine  X  X  X  X  X 
the  presence  of  appropriate  and 
current  versions  of  standards  and 
specifications  and  to  check  tailoring 
against  requirements  for  valid 
administration  of  contract. 
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29.  Prepare  and  review  sample  contract  to  X  XX 

ensure  inclusion  of  necessary  clauses 
and  forms,  deletion  of  unnecessary 
forms  and  clauses,  proper  cross 
references  to  MIL-specs  included  in 
SOW/CDRL,  and  other  technical  and 
clerical  details. 


51.  Develop  an  internal  CAS  plan  for  X  X  X  X 

assignment  of  manpower,  test  and 
review  participation,  inspection 
points,  and  communication  responsi- 
bil ities . 


54.  Ongoing  evaluation  and  review  of  XX 

contractor’s  management  system  for 

software  development. 

55.  Monitor  critical  subcontracts  for  X  X  X  X  X 

flow-down  of  software  contract 

requirements. 


56.  Monitor  cost,  schedule,  and  perform-  X 
ance  of  software  development  tasks. 


X  X  X  X 
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57.  Resolve  technical  and  management  x  XX 

problems  in  software  development 
process. 

59.  Review  contract  modifications  for  any  X  X  X  X  X 
possible  effects  on  monitoring 
software  development. 


60.  Monitor  development  and  deliveries  of  X  X  X  X  X 
software  Contract  Data  Requirements 
List  documents. 


61.  Monitor  and  review  compliance  with  X  X  X  X  X 

automated  test  system  standards  and 
documentation. 


62.  Audit  contractor  compliance  with  x  X 

software  requirements  for  configur¬ 
ation  management  and  quality 
assurance. 


64.  Follow  up  all  action  items  designated  X 
by  SPO  as  a  result  of  each  formal 


X  X  X  X 


review. 
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68.  Review  the  Computer  Software  Config-  X  X  X  X  X 
uration  Item’s  functional,  perform¬ 
ance,  data  base,  qualification, 
interface,  and  delivery  requirements 
through  the  formal  Software  Specifi¬ 
cation  Review. 


69.  Assess  proceeding  with  contractor’s 
designs  for  each  Computer  Software 
Configuration  Item  through  the  formal 
Preliminary  Design  Review. 

71.  Monitor  software  coding  phase. 

72.  Monitor  software  integration  and 
test. 
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76.  Document  the  completeness  of  meeting  X  X  X  X  X 
requirements  and  specifications,  and 

testing  of  each  Computer  Software 
Configuration  Item  and  Software 
Product  through  formal  Functional  and 
Physical  Configuration  Audits. 

77.  Monitor  planning  and  execution  of  x  X  X 

Independent  Verification  and  Valid¬ 
ation. 


78.  Monitor  development  of  automated  X 

tools  used  to  prepare  technical 
manuals,  operating  procedures, 
engineering  drawings,  etc.,  to  assure 
compatibility  with  end-user  systems. 

79.  Monitor  assurance  that  configuration  X 
management  and  historical  data 

systems  are  maintained. 


X  X 


X  X 
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80.  Determine  capability  of  maintenance  X  XX 

procedures  to  human  factors  and 
equipment. 


81.  Evaluate  training  requirements  for  X  XX 

operation  and  support  of  software 
system. 


84.  Complete  software  acceptance  pro¬ 
cedures. 


X  X  X  X  X 


86.  Review  the  Software  Quality  Assurance  X  X 

Plan  for  postdevelopment  software 

support. 

87.  Review  Software  Test  Plans  for  post-  XX  X 

development  software  support. 

88.  Determine  that  proposed  engineering  XXX 
changes,  deficiencies,  and  latest 

defects  in  software  are  corrected  by 
the  contractor  and  the  Configuration 
Management  Plan  adjusted  accordingly. 
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89.  Establish  a  Configuration  Management  XXX 
Plan  for  postdevelopment  software 
support. 
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0 


Dennis  Blosser  &  Associates 
Final  Report 
Page  Appendix  D-l 


Lt.  Arnold 
Information  Systems 
Norton  AFB 

Don  Brautigam,  Industrial  Engineer 
MCCR  Rep.,  Production  Operations  Division 
Rockwell-Anaheim  AFPRO 

Lt.  Crawford 

Information  Systems  and  Computer  Resources 
B.M.C.,  Norton  AFB 

Tom  Ferris 
QA  Specialist,  QAX 
Rockwell-Ananeim  AFPRO 

Bernard  Katz 
Aerospace  Corporation 
DSP  Program  Office 
Space  Division 

Maj .  Mitchell 
Information  Systems 
B.M.C.,  Norton  AFB 

Myron  Pidhany,  Electrical  Engineer 
MCCR  Focal  Point,  EPP 
Rockwell-Anaheim  AFPRO 

Shiguru  Shozi 

QA  Manager  for  Configuration  Mgmt. 

DSP  Program  Office 
Space  Division 

Janice  Smith 

Deputy  Director,  Mission  Control 
Mil-Star  Program  Office 
Space  Division 

Leona  Solar 

ACO/TMD,  Contracts  Division 
Rockwell-Anaheim  AFPRO 

Lt.  Col.  G.  Stojanowsky 

Information  Systems  and  Computer  Resources 
B.M.C.,  Norton  AFB 
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Mark  L'Ecuyer 

Mfg.  Engineer/MCCR  Focal  Point  for  PD 
AVCO  AFPRO 

Captain  Jeffrey  Ford 
MCCR  Focal  Point 
Lockheed  AFPRO 

Gerald  Conroy 
Chief,  Pricing 

Software  Evaluation  Team  Member 
Lockheed  AFPRO 

Captain  Ruben  Lopez 
Chief,  Subcontract  Division 
Software  Evaluation  Team  Member 
Lockheed  AFPRO 

Don  Lewis 

Industrial  Engineer 

Software  Evaluation  Team  Member 

Lockheed  AFPRO 

Colonel  Bob  Peters 
EP  Division  Chief 
Lockheed  AFPRO 

Captain  Chuck  Davidson 

Computer  Resource  Acquisition  Engineer 

Eglin  AFB 

Major  Bill  Miller 

Systems  and  Integration  Branch  Chief/EN 
General  Dynamics  AFPRO 

Gary  Gaston 

Electronics  Engineer/Program  Manager 
General  Dynamics  AFPRO 

Jim  McQuaid 

QA  Software/Electronics  Specialist 
General  Dynamics  AFPRO 

Sam  Horstman 

QA  Technical  Section  Chief 
General  Dynamics  AFPRO 
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Bill  Turner 

QA  Software  Specialist 
General  Dynamics  AFPRO 

Lynn  Jones 

QA  Software  Specialist/Focal  Point 
General  Dynamics  AFPRO 

Captain  Terri  Fox  Daeke 
SPO  Software  Support 
Hanscomb  AFB 

Steve  0 1 Shaughnessy 

Mitre  Corporation 

SPO  Software  Manager/Peace  Shield 

Lieutenant  John  Gann 
MCCR  Software  Focal  Point 
AVCO  AFPRO 

James  Wheeler 
QA  Software  Specialist 
AVCO  AFPRO 

Muffy  Staffieri 

MCCR  Software  Focal  Point  for  Subcontracts 
AVCO  AFPRO 

Ma j .  Stroud 
Information  Systems 
B.M.C.,  Norton  AFB 

Darrah  Whitlock 

QA  Specialist,  Plans  &  Eval.  Branch 
Rockwell-Anaheim  AFPRO 

Lt.  Col.  Barry  Prins 

HQ/AFCMD 

Kirtland  AFB 

Stan  Brown 
HQ/AFCMD 
Kirtland  AFB 

Mike  Bates 
HQ/AFCMD 
Kirtland  AFB 
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Col.  Martin 
HQ/AFCMD 
Kirtland  AFB 

Col.  Williams 
HQ/AFCMD 
Kirtland  AFB 

Lt.  Col.  Ed  Stevens 
Space  Division 

Donna  Mazzanti 
HQ/AFCMD 
Kirtland  AFB 

Dennis  Drouilland 
HQ/AFCMD 
Kirtland  AFB 

Philip  Babel 
ASD 

Wright-Patterson  AFB 

Capt.  Dennis  Smith 
AFBRMC 

Wright-Patterson  AFB 

Lowell  Simon 
Martin-Marietta  AFPRO 
Denver 

A1  Chmara 

Martin-Marietta  AFPRO 
Denver 

Andy  Serino 
Martin-Marietta  AFPRO 
Denver 

Ken  Hackett 
Martin-Marietta  AFPRO 
Denver 

Maj  Hutchinson 
HQ/AFCMD 
Kirtland  AFB 


